Tweet from Virtbiz:

VIRTBIZ

 Cogent is currently experiencing an outage. VIRTBIZ is routing around this now. Some Cogent customers will be experiencing temporary issues.

Also updating customers on Twitter.

Edit: 8:45am – all virtual servers are back online.  Still investigating root cause of this outage.

Edit 9am – The node’s networking took a dive right after generating a status report email detailing among other things networking traffic. After failing to respond, the PDU that monitors the server rebooted the server by power cycling it.  I’m be looking into the possibility of the node monitoring it’s own networking and doing a clean shutdown & reboot should networking be lost. However, it will take some work to out timing of when the PDU should take over. Also looking to see if there are updates to the programs used inside the status report generation scripts.  Though a problem with the script or the programs used by it is somewhat unlikely. We use the same scripts on all of the nodes, which have uptimes as long as 375 days… GT24-3 itself was up 99 days before this happened.

Regarding the unexpected shutdown of your equipment on 8/4/2011, here
are the results of our investigation as well as the proposed remedy.
Forgive the length of the report.  I feel it's best to give a full
picture of what occurred so that you can see the context.  As is often
the case, the issue was really the result of several small, seemingly
unrelated issues that would otherwise be quite harmless, until they come
together to create a "perfect storm".

Beginning on 8/1, we have had extensive work performed all week on the
cooling systems in the datacenter in order to increase cooling capacity.
 With our continued growth and relentless extreme heatwave that Texas
has been experiencing for the past month, our cooling systems have been
working non-stop.   We made the decision in early July to expedite the
planned installation of additional cooling, which had originally been
scheduled for Fall 2011.  We ordered the equipment and worked with our
contractor to begin work ASAP, even in their peak season.

The new equipment was commissioned into service yesterday afternoon
around 1:30pm and we immediately began reconfiguring the overall cooling
strategy in the datacenter to make efficient use of the additional
capacity.

Beginning 3:30, we were alerted by ERCOT, the power grid operator for
the State of Texas, that they would begin shedding load on the Texas
grid because it had become overloaded.  We "changed gears" from the
cooling project to the more immediate electric issue.  At 3:43pm, we
transferred the datacenter to generator power.  (That event ended at
6:16pm.)

Around 9:00pm, the new cooling system went offline.  We did not process
this as a critical alarm since without the new unit we are still within
adequate cooling capacity for continued operations.  However, we did not
fully complete our changes to the facility's cooling strategy when we
received the ERCOT notification.

At 10:08pm, the NOC processed 2 alarms: a power room temperature alarm
and an inactive switch alarm.  An on-site tech was immediately
dispatched to investigate and remedy.  Also, as is policy, a senior tech
was called to the datacenter and I was also called out.

We found that a sub-panel distribution breaker had been tripped.  This
was not due to over-current (overloading) of the circuit because we are
reading load at just over 60% rated capacity.  It was also not due to
any line-fault.  We have made the determination that the breaker tripped
due to a faulty thermal protection mechanism.  In other words, the
circuit breaker appears to be faulty.

Circuit breakers work on the theory of thermal-magnetic protection
whereby if they reach a certain threshold, they will trip and disconnect
the circuit.  This hasn't typically been an issue since we keep the
datacenter and the power room well-regulated with regard to temperature.
 With the increase in ambient temperature, the weakness on the breaker
was exposed.

Here is the 3-part remedy:
1) Work that was put-off due to the ERCOT event has been completed on
backup cooling arrangements for the power room.

2) Our air conditioning contractor is on-site to diagnose and remedy the
cause of failure for the new capacity.

3) We will make plans to change the weak circuit breaker.  If it is
necessary to institute a planned shutdown for this to happen, then you
will be notified in advance so that you will be aware, and may
participate by issuing a graceful shutdown to your equipment as needed.

Of course, my team and I very much regret that your operations have been
impacted by this issue.  We take a fair amount of pride in our work and
always strive to provide the best level of service possible.  Of course,
we are human, and humans are often, sadly, imperfect.   I'm sorry that
our imperfection has caused an issue for you.

Please let me know if I can answer any questions, or if there is
something further that I can do to assist on this.  Thanks for your
business and patience.  I appreciate the continued opportunity to serve you.

Update:

There was, as the A/C techs are installing a new 20 ton Liebert and they have been in the 480 panel. They seem to have bumped a breaker which was reset. 

The SR1435 did not come up and I booted it just a minute ago.

Thanks for the heads-up on this, and all should be well now.

Let me know,
Jack B. – VIRTBIZ Internet Support

Here’s Virtbiz’s response to my inquiry:

The VIRTBIZ network is experiencing a DDoS in excess of 2Gbps which is
causing network impairments for some customers. We are sorry for this
issue and have network engineers working to mitigate this now. At this
time, I believe network connectivity should be restored to normal for
you but please advise if you are still experiencing difficulty.

Tks,
Stuart G. - Support Team
VIRTBIZ Internet Services

We are experiencing intermittent network performance issues to all servers in the Dallas DC. The servers themselves are up and running. From an end-user standpoint, it will appear as a very slow connection or even timeouts. We have a trouble ticket in with the DC and will update as we know more.

Update:  The DC has identified an IP address from India that was attempting a large number of connections. This IP has been firewalled out.

It’s all back… it appears to have been a network hiccup somewhere. None of the servers show any downtime.

Update: 3:15pm – Updates are completed and the node has been restarted