26-Jan-2014

Moving production

After months of setup, we finally successfully relocated the main machines that support our ERP application to the new data centre today.

We actually attempted this back in October, but we had to roll back after a very strange hardware issue that didn't appear to be a hardware issue. A mezzanine fibrechannel card reported all in the green, but irregardless of what we tried, failed to recognise the disks. After fighting this with everything we could think of (array setup, FC switch setup and zoning, Virtual Connect setup, host hardware, first and second level support from HP), and compounded by some poor network support from the CoLo, we rolled back.

When we got back to the original data centre, and after talking to HP third level support, it was realised that the fibrechannel card was showing bizarre numbers for the WWNs. We deleted the Virtual Connect profile associated with the blade, and recreated it, and while the WWNs now looked appropriate, the card would still not see the drives. In desperation, we replaced the card, and everything came good. Everyone breathed a huge sigh of relief.

On reflection, we should have suspected the card was dodgy before we rolled back, as we had HP engineers on site with spares in case anything like this actually happened.

When we came to do the move again today (this time armed with the offline diagnostics tool, just in case), everything went much more smoothly. Let's hope we have no bedding down issues when the full production load comes on on Tuesday morning.

Posted at January 26, 2014 11:28 PM
Tag Set:

Comments are closed