SAN problems
I've been a little quiet on the blog here of late as we have been struggling with a number of problems, the first of which appears to be a firmware interaction issue on our SAN.
We are running the following firmware:
| Device | Description | Firmware |
|---|
| HSG80 | StorageWorks array controller | ACS 8.8-4 |
| Silkworm 3200 | Brocade 2Gb 16 port FC switch | 3.2.0a |
| Silkworm 2800 | Brocase 1Gb 16 port FC switch | 2.6.2d |
| Emulex L9000 | 2Gb FC host bus adapter | CS3.92A2 |
| Emulex L8000 | 1Gb FC host bus adapter | DS3.92A2 |
These revisions are the latest and greatest at this time.
Additionally, we are running OpenVMS 7.3-2, with FIBRE_SCSI-V0400 on (more on this in a minute).
The problem we are seeing is that not all fibrechannel paths are available to all hosts in the cluster. By this I mean we expect to see four paths in existance and available (as the HSG80s have 4 ports) but we only see three in existance (but some machines do show four).
We strongly suspect a firmware issue due to the fact that we have the problem on both (independant) fibrechannel fabrics and across multiple physical machines. Interestingly, not all machines in the cluster have the problem, so I suppose it is conceivable that the issue is a host based code issue. However, again, read on.
Our second problem is that we applied the FIBRE_SCSI-V0700 patch at the recommendation of HP, only to find that it caused CPUSPINWAIT crashes on partitions in our GS160 AlphaServers. Note that while the patch was on, we were still experiencing the path problem, which seems to rule out a host based issue. Needless to say, we removed this patch, and actually thanked the Engineering Team for coming up with $ PRODUCT UNDO.
Information on this crash that we have been able to glean out of Engineering indicates that FIBRE_SCSI-V0700 is aggravating a known issue with IO_ROUTINES.EXE. As more I/O is done, non-paged pool is allocated from the lookaside lists in the RAD where the I/O takes place, eventually exhausting them. When this occurs, the operating system attempts to allocate non-paged pool from the other RAD, and when the request is not fulfulled quickly enough, it crashes the machine.
The reason this machine crashes is that it's the one doing the vast majority of our backups, and so is performing the most I/O in parallel.
Before we removed the patch, we tried things like playing with SYSGEN parameters NPAG_GENTLE and NPAG_AGGRESSIVE in an attempt to reclaim pool, and also attempted allocating more non-paged pool in the RAD that (apparently) was always exhausting it (we found out that it can occur in either RAD).
If this isn't enough, we have a recently modified user mode image that is crashing a machine in the same cluster on a reasonably regular basis. All the image does is use RMS to extract some information from some indexed files and write a report. However, every second or third night, the machine that is running it will crash with a system service exception. Note again we have had crashes on multiple machines so this looks like a latent bug in the operating system.
We are certainly keeping Engineering busy...
Posted at December 18, 2005 6:12 PM
