… to add the zip module at a customer’s request. Shouldn’t be long… ~ 1 hour. Most of the time the server will be available… occassionally WHM will take apache offline a couple of times.
1:30 pm – Done
Archive for June, 2009
… while we resize the filesystem for the new server. It shouldn’t be offline for long, expected to be less than 30 minutes.
Step 1 – Stop container – done
Step 2 – Unmount filesystem – done
Step 3 – check fs – Done
Step 4 – Resize filesystem – done
Step 5 - Resize logical volume – done
Step 6 – Remount filesystem – done
Step 7 – Restart container – done
Step 8 – Relax — working on it…
To our VPS customers,
Since the troubles with HyperVM/LXAdmin, we’ve been busy looking at alternatives. By far the most promising is turning out to be FluidVM from a company called Binary Karma. Many of the features are very similar, and some are signifigant improvements, such as the ability to use iSCSI/SAN storage as drives for virtual machines. This is something we’ve wanted to move to anyway. And while technically it was possible with HyperVM, it wasn’t as integrated into the environment.
Backup procedures will be changing. All the Compute Nodes need to be reformatted anyway, and LVM installed on a couple that it wasn’t before. This will enable us to do backups without interrupting the virtual machines.
Dallas servers are going to be first to migrate to the new environment. Existing virtual machines will be migrated to new servers with little downtime. Occassionally, we do see issues with quotas and cpanel licensing but in both cases the fixes so far have been quick and easy.
Atlanta will take a little longer and will require some downtime while the OS gets re-installed and vm’s restored. Customers wishing to migrate to Dallas will be given the option to do so beforehand. Though ip address changes will have to happen to migrate. A date has not been set for this to occur…
All VPS customers hypervm/fluidvm, will receive new instructions with login details as they’re migrated.
Jun
16
We’re doing a live migrate on cpwhm1… this isn’t changing platforms or anything else just moving nodes. This is to clear off a node to begin the process of moving to the new platform. We need a couple of clean nodes to work with on testing, but we’re excited to find and test out Binary Karma’s FluidVM product. In some ways their product is better aligned with where we want to go anyway… with things like iSCSI storage (hint: think fault tolerance).
The VM should only be unavailable for a few minutes during the process.
The live migrate went smoothly… cpwhm1 virtual server was offline from 6:19p to 6:24p. Re-applied cpanel license 1 minute later… All looks well…
You may have heard, the future of our provider of our virtualization management software, LXLabs, is in serious doubt at this point. The founder and developer has taken his own life, and there are several known vulernabilities in the software that have been made public recently. Some were fixed others remain in question. In response, we are have taken down the HyperVM service as well as LXAdmin/Kloxo on all our company vps’es. For customers running LXAdmin we strongly suggest doing the same.
Here’s how:
(log in with ssh as root to your vps)
service lxadmin stop (stops the panel from working temporarily)
chkconfig –level 2 lxadmin off
chkconfig –level 3 lxadmin off
chkconfig –level 4 lxadmin off
chkconfig –level 5 lxadmin off (these turn off the service for those runlevels)
chkconfig –list lxadmin (shows which level are ‘on’ … should all be off now)
(log out)
If there is something you need the panel for, we strongly recommend not leaving it running. Turn it on (service lxadmin start), do what you need, and then turn it back off (service lxadmin stop).
However this is not a long term solution. We are looking at the available virtualization solutions, and trying to work in which would provide a stable environment (something LXLabs’ needed improving on anyway), painless migration (as much as possible), and growth potential. We are heavily leaning to one of the more mature open source products. As this potentially gives us peer review of code asn actual access to the code itself. I am personally a 25 year veteran of software development, so I just feel better when I can get my hands on the code. Of course, we’ll do everything possible to ease migration pains when the time comes. But as we’re all about open communications with customers, we wanted to let everyone know the direction we’re heading and get your input if you have experience good or bad with a virtualization management platform. We’d love to hear your thoughts on the matter.
Customers wanting to come off the LXAdmin control panel to something else, please contact us. We can offer several alternatives – cp/whm, directadmin, dtc, ispconfig, etc. The only one we will charge for is cp/whm … namely because we can’t buy that product only lease it at $15/mo. We will provide a cp/whm license to you at our cost if that’s the panel you desire. For DirectAdmin we will buy an internal license (ie stays with us if you leave) for your use for free as long as you’re with us. Some panels do not like being placed on an existing configured server. For those installs we’ll be happy to provide a second vps configured with the new panel free for a month in order to move accounts over to the new environment. Please feel free contact us to discuss your situation and how we can help.
Robert
Lagniappe Internet LLC
cPanel/WHM is not liking the kernel change after the move. It was running however when it went to update cPanel/WHM errors started showing up. We’re restoring the previous environment, and will be moving the data over shortly. During the data move the virtual server including website will not be available (estimated @ 15-30 minutes).
The new plan to migrate to the new server will be to bring a new virtual server up, and then migrate accounts over to the new server one at a time.
22:56 – Data migration is completed. CP/WHM being reconfigured… so it may be flap up/down for the next 30 – 60 minutes. Will post when done.
23:21 — CP/WHM is back up.
Later on this evening we will be migrating the cpwhm1 virtual server to the GT24-2 node. During that time, the virtual server will be down on 2 occassions. The first to take a final backup in preparation for the move and then again during the move. The first downtime should be short … less than 5 minutes based on previous backups. The other will be a little longer – between 30 and 60 minutes based on previous migrations. All ip addresses and account specifics should move with the vps.
The new server features dual quad-core processors, 32GB of Registered ECC RAM, 4 Western Digitial RE2 hard drives connected to a LSI/Dell Perc 5si RAID controller with 128MB of dedicated RAM and battery unit… This is a signifigant upgrade for this vps. On top of the increased hardware, the new server utilized logical volume snapshotting for backups - meaning no nightly downtimes for backups (even though they were for only about 3 minutes… none is still better!)
This post will be updated as the migration progesses…
8:30 – vzDump begun
9:20 – 1st vps shutdown from vzDump 2nd sync begun
9:22 – vps back up – waiting for tar.gz to complete
9:50 – backup complete – validating file
10:15 – begining migration to gt24-2
11:17 – migration completed succesfully!!
11:30 — going to do a quick reboot to reconfigure memory allocations (increase it by 4x)
11:35 – Done…
We just received this email from LXLabs:
Dear Customers,
Multiple security vulnerabilities were discovered in hyperVM and Lxadmin/Kloxo. It is recommended that you update your hyperVM/Kloxo systems to the latest version, as soon as possible.
Details of the vulnerabilities will be posted in the coming days in our forum.
On hyperVM or Kloxo master, Run:
/script/upcp
Lxlabs Support Team
We are updating the HyperVM system as I write this email… Please be sure to run your update. If you need help or would like us to do it for you, simply put in a ticket at http://www.lagniappeinternet.com/members/ and we would be happy to do so.
Lagniappe Internet
In an effort to further stress test the new server, we’re moving our own site there… This will result in a brief downtime while the vps is migrated. Should be back up in 30 minutes or so.
Jun
2
As sometimes happens with new equipment, it fails… Even to the best equipment. This wasn’t a drive you’d buy at the local office supply or big box electrnics store. This was one of the 4 Western Digital Enterprise class RAID Edition 2 (RE2) drives that failed shortly after being placed in the rack and left to burn in… This is exactly why we let servers sit for a while before using them. No customer data had been placed on the server. Our own website was placed there for a short while, but when the node experienced problems we moved it back to the other server – giving us some real world experience in a problem situation live migration.
After working with it a bit and determining the drive itself was at fault a replacement drive was ordered. The existing drive will be swapped out under WD’s warranty terms. While the node is operational, with no RAID mirror protecting the data, we chose to not place any customers onto the server. So for the time being XEN is not available as an option when ordering a VPS. If you really want XEN and need to sign up today, we can temporarily place you on an OpenVZ node in the same d.c. Once the server is back to a trusted state, we’ll migrate your vps over to the server – but this will result in a small downtime during the migration.
The replacement drive is in FedEx’s hands, and we expect the drive to be delivered tomorow. After it’s in, the drives sync’ed (up to 24 hours), and stress tested, we expect the server to be available for use by the coming weekend.
Edit 6/3: The new drive did in fact arrive today and was the defective replaced. The controller is in the process of rebuilding the array as I type this. Once it is done, I’ll start a process of stressing the drives just to be confident in them before opening the server up to customers… We wouldn’t have normally been as relaxed in the replacement process. But since no customers sites are hosted on the server, we thought it better to take the time and be sure it’s 100% up to the job for which it’s designed.
Comments Off