SANS

Written in 1999, just when FC was starting to take off

Overview

Fibre Channel is an ANSI standard network and storage interconnect system employing fast switched fabrics over fibre optic connections. The ANSI standard specifies the use of Fibre Channel hubs and switches. However, OpenVMS currently only supports switches. OpenVMS support for Fibre Channel was introduced in version 7.2, and immediately provided the following main features:

Configurations

OpenVMS currently supports four switch fabric configurations, allowing users to scale storage to multi-terabyte capacity with ease.

Single switches

Technically not a "fabric", this is generally the starting point for people "getting their feet wet" with FC configurations in the OpenVMS world. For obvious reasons, there should be dual independent fabrics that are mirrors of each other. This implies the entry level setup is two single switches.

Cascading switches

Connecting switches to each other with interswitch links enables the fabric to grow on demand. Cascaded fabrics are switches connected to each other in a "line" configuration. This setup is useful for up to a few switches. The main disadvantage of cascading is that as the fabric grows beyond say, three switches, the multiple hops between switches to get from any point in the fabric to any other point begins to present a problem.

Rings

To reduce the effect of multiple hops on performance, a ring can be used. This in effect joins the ends of the line of switches together. There are certain limitations with this configuration too, but at least the number of hops is potentially half the longest path in a cascade.

Modified Mesh

Mesh fabrics are the best fit for up to about six switches. Switches are connected to each of their neighbours in a two dimensional grid (in general, two rows of three is about the limit). This is a nice fabric to use as a stepping stone to the best design...

Trees

Trees are the ultimate in fabric flexibility. In a tree fabric, you have core switches and edge switches. The core switches have nothing but other switches connected to them. And the edge switches have the storage and hosts connected. If this is done correctly, there should never be more than two hops to everywhere in the fabric.

If more bandwidth is required, a "fat tree" model can be used. This is achieved by adding additional ISLs between the edge and the core switches.

Large numbers of ports in the core make things much easier for large fabrics. You can however use multiple smaller switches.

My experiences

Starting out

I started with a single 16 port switch on a development cluster to get an idea of how to set SANS up. When I first deployed this system in a production environment, due to budget constraints, I could only purchase three switches. That was not conducive to setting up two redundant fabrics.

So I created three fabrics. With the addition of two more HBAs (which cost a lot less than another switch), I was able to wire it up with the disk controllers split between switches one and two, one and three, or two and three.

Management Nightmares

Of course, three fabrics (four counting the single switch I had on development) were a pain to manage. Not at the beginning, but as the company quickly began seeing the benefits of Fibre Channel, they started wanting to convert everything. This was fine by me, but first we had to get at least one more switch in to make two fabrics of two switches each.

Growing. And Growing...

This was the first reorganisation I did. I ended up with two cascaded switches in each fabric. Each switch was connected to the other with two ISLs for redundancy. And we were off and running. Shortly thereafter, I was able to obtain four more switches, and I quickly converted to a modified mesh design with four switches in each fabric. The lesson I learned in this reconfiguration was that you really can do reconfigurations on one fabric while the other fabric maintains all your disk connectivity.

When I reorganised into a modified mesh design, I also took the opportunity to move all the switches that support the second fabric into a completely separate cabinet. This gives me four separate power supplies (two per fabric) and physically separates the cables coming into the two fabrics. Now there is no chance of someone pulling a cable on the wrong fabric.

Recently, I have managed to get my hands on four more 2GB switches. The existing fabrics are made up of 1GB 16 port switches. The new faster switches will be used to construct a skinny tree core/edge design. And hopefully that will be the last major reorganisation I will ever have to do at this site.

Conclusions

Storage Area Networks based on Fibre Channel is the way to deploy disk storage on OpenVMS at the moment. It's expandable, flexible, and extremely easy to manage once initially set up.