We’re doing some quick patching and reboots to address a Linux kernel vulnerability that has been announced. The servers should be down less than 5 minutes and virtual servers they house less than 20 minutes. Sorry for the inconvenience but it
Archive for the ‘Promotions’ Category
We received notice from Virtbiz that they will need to cut the power off tomorrow night at 11pm Central time.
[quote]As you should be aware from our previous announcements, we have recently replaced a primary UPS that provides filtered power to your equipment. The change took place because of certain errors that could not be cleared from the replaced unit. Accordingly, a new system was ordered and installed over the weekend. This was done without interruption to service.
Unfortunately, due to technical limitations of the outgoing maintenance bypass and the new UPS, we are forced to take a small outage in order to transfer back to fully protected power. In order to limit possible exposure to failure from utility power service, we have elected to take this emergency maintenance window as fast as practically possible.
We are scheduling the emergency maintenance window for Tuesday, September 21 2010, after 11:00PM Central Time.
Customers located in Rows 10, 11 & 12 will experience a brief power outage while power is switched to the new UPS. The interruption will last no longer than 1 minute, and may in fact be contained to under 5 seconds. Of course, that will likely be long enough to cause an outage to your connected equipment.
…
[/quote]
Sep
5
We’re finalizing a Xen VPS server. It’s in Dallas – has 2 Quad Core AMD Opteron processors, 16GB of ECC RAM and 4x 750 WD Raid Edition (RE2) Enterprise class drives connected to a Dell/LSI PERC RAID controller in HARDWARE RAID-10 for speed and reliability.
Xen VPSes have a greater isolation level than OpenVZ and allow other Operating Systems to be run – like Windows. We will be allowing Windows VPSes, but you will need to provide an ISO of the OS version you want (will not count to your disk space and will remain on the server as a CD/DVD drive) and a scan of the License key.
Pricing will be announced shortly. Contact us if you are interested in a Xen VPS.
We now have a cPanel/WHM reseller vServer available in Atlanta. Same prices as Dallas, but use this link to sign up: http://www.lagniappeinternet.com/members/signup.php?clienttype=16
Aug
28
This is the latest update (original at the bottom):
Subject: [#MAU-767967]: Network outages – still happening
As a follow-up, I would like to share with you a postmortem of the
routing event you experienced this morning.
At about 6:30AM we received a preliminary alarm of a failure on one of
our customer gateway routers. The event was logged and the system was
enabled for failover to the redundant processing card on that gateway
router. A short time later, the routing fabric of the gateway failed
and caused a disruption in network service. Our technician brought the
router back up and ran a diagnostic, which ran clear. At that time, the
system was brought back online and routing resumed to your equipment.
After running stably for about 30 minutes, the system and its backup
failed again. Senior technicians implemented a hard-failure plan and
brought up a “warm spare” gateway router and began loading the
configuration. Network service was restored individually to each
customer affected by the outage as the configuration was checked-in and
loaded.
I should note that while you are one of several customers that was
affected by this incident, this was not a full-scale routing outage.
Our network architecture makes extensive use of sandboxing in order to
not put all eggs in one basket. Nevertheless, I understand that while
not everybody was affected, YOU were affected, and I do apologize for
the incident.
At this time, our warm spare has been placed into production and
functioning normally. We have brought in a cold spare and activated it
into standby so that redundancy is still in place. We anticipate that
we will replace the affected gateway router with new hardware. When
that happens, we will migrate your routing onto the new system, place
the warm spare back into standby and pull power from the cold spare.
All this will be seamless and will go unnoticed from a connectivity
standpoint.
Please be assured that your VIRTBIZ team will continue to review this
incident so that we can further improve our service to you.
I hope that you have a pleasant remainder of your weekend.
Original posts and updates:
We are experiencing a network issue between cogent and virtbiz this morning. First shortly after 6am, and lasted apx. 2 minutes. By the time I started to look into it and contact the DC, it was back up.
It is occurring again now. This time I was already on the servers looking into a spam report from a VPS account. Check checks showed the link between cogent and virtbiz to be down. It back up to the point I could send virtbiz a message (sure they already knew but just in case…), and am waiting to hear back. It had come back up at 7:50. Actually the virtbiz support site came up about 10 minutes before that.
At 8:01, it’s out again… Still waiting on an answer from vb…
As you can see by this, it’s having a problem finding a route.
[root@gt24-1 ~]# traceroute support.virtbiz.com
traceroute to support.virtbiz.com (208.77.216.244), 30 hops max, 40 byte packets
1 208.75.228.193 (208.75.228.193) 0.482 ms 0.809 ms 0.952 ms
2 tulip-core-2-ge3-8.tshost.com (208.75.224.5) 0.326 ms 0.318 ms 0.335 ms
3 core-1-gi7-2.tshost.com (208.75.224.13) 0.314 ms 0.344 ms 0.380 ms
4 te8-3.mpd01.atl01.atlas.cogentco.com (38.104.182.45) 0.310 ms 0.274 ms 0.287 ms
5 te0-0-0-6.mpd21.iah01.atlas.cogentco.com (154.54.28.254) 14.704 ms 14.816 ms te0-2-0-1.mpd21.iah01.atlas.cogentco.com (154.54.2.146) 14.697 ms
6 te2-1.mpd01.dfw01.atlas.cogentco.com (154.54.5.133) 20.387 ms 20.471 ms te3-4.mpd01.dfw01.atlas.cogentco.com (154.54.25.93) 20.560 ms
7 vl3834.na01.b000868-0.dfw01.atlas.cogentco.com (38.112.35.54) 21.105 ms 21.519 ms vl3534.na01.b000868-0.dfw01.atlas.cogentco.com (66.250.13.178) 20.402 ms
8 38.107.227.210 (38.107.227.210) 20.385 ms 20.646 ms 20.818 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 25.759 ms 25.751 ms 25.755 ms
16 * * *
17 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 25.950 ms 25.892 ms 25.902 ms
18 * * *
19 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 26.025 ms 25.996 ms 26.005 ms
20 * * *
21 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 26.172 ms 26.126 ms 26.668 ms
22 * * *
23 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 26.741 ms 26.330 ms 26.374 ms
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
As of 9am, we are back up at the moment… Just received this from VB:
We have been having an issue with one of our routers. Our technicians are correcting the issue and services should be fully restored shortly.
I’ll close this ticket now. If we can be of further service, just respond to this message.
Thank you
Jack B. – VIRTBIZ Internet Support
It’s upgrade time again… Time to retire one of the older servers. The new server is another Tyan GT24. We’ve been really pleased with them. This one will have 2 Quad Core AMD Opteron processors, 16GB of RAM, and 4x drives in RAID-10. Can’t say what size drives will be yet. At least 500GB each, likely to be 750′s or 1TB. And yes, they will be on a PERC hardware RAID controller.
The plan right now is to replace the IBM X335 server. We expect this will be sometime in September – so we don’t rush things. We’d rather see it done properly than quickly. Since all public facing servers are virtualized, downtime should be able to be measured in seconds while the VM migrates.
We’ve asked Virtbiz to look into the issue and are waiting on word back.
Early the morning, GT24-2 experienced an issue, we remotely reset the machine and brought it back up. About an hour later, it happened a second time. The about 1pm we lost connectivity to all equipment there. Servers are coming back online and it appears they had lost power.
Edit -
Virtbiz says they had an issue with the power to the rack and had to shutdown the power temporarily.
We have migrated the cpwhm1 (reseller cpanel) virtual server to a faster node… The new node will have access to faster drives, more memory and additional cpu cores.
Mar
2
The gt24-2 node had an issue starting at apx 4:20am Central time. The load started spiking and the node and all vm’s started slowing down, until the point they became unresponsive. We’ve run updates on the OS and are still investigating what happened.
As the /var/log/messages shows nothing out of the ordinary up until it stopped, we’ll also be logged into the node to see if we can see what is occurring in real-time. It also means it the delay until notification will be gone. We’ll also be sending out a replacement server to the DC in case we need to migrate the VMs off of it. If it does happen again we’ll migrate a couple VMs off the server as well, but trying not to as the server they will be migrated to has less RAM, CPU and disk to begin with. If we do it’s a stop gap measure while the replacement is setup.
Lagniappe Internet’s founder Robert Porter has passed the Certified Internet Web Professional, aka CIW, Database Design Specialist exam today.
CIW descibes this exam…
CIW v5 Database Design Specialists have mastered the knowledge and theory of database design that applies to the most popular database platforms. These professionals help solve the problem of poorly designed databases. Aimed at database programmers and administrators alike, this vendor-neutral certification focuses on universal database design principles and SQL. The CIW v5 Database Design Specialist exam validates foundational knowledge of databases in general, such as Oracle, IBM, DB2, MySQL and others.
CIW v5 Database Design Specialist certification is valuable for individuals working in fields such as IT, database development, application development and other areas that depend on Web-enabled systems for productivity. To become a CIW Database Design Specialist, the candidate must pass one required CIW exam AND complete the CIW Certification Agreement by logging in to the CIW Candidate Information Center.
Together with the previously passed CIW Associate requirements, gives Robert the “CIW Professional” certification as well. This is in a long list of certifications gained over the years including A+, Network+, Security+, MCSE+Inet, and CCNA.
Comments Off