e i g h t - c u b e d . c o m 

Directory index cache

Over the last few weeks, I have been dealing with the performance fallout from an application on the main production cluster exceeding a "wise" number of files in a single directory.

To improve directory lookup performance the file system maintains a cache of directory entries in memory. As this cache data structure is limited in size, the algorithm that maintains the cache reduces the size of the filename samples as the number of entries in a directory grows. Normally this is an acceptable strategy. However, when all the filenames start with the same few characters, and the number of files exceeds 650,000 (in this case), the directory index cache becomes ineffective, and the file system has to scan the directory sequentially to locate files.

When this happens, and your directory file is over 61,000 blocks, you get degraded performance (to use the technical term).

Interestingly, this problem manifested as complete cluster hangs upwards of a minute. The hangs were severe enough that performance statistics usually gathered by Polycenter Performance Advisor were either not being collected, or were not being written out to disk. This made diagnosis rather difficult after the fact. I didn't particularly want to camp out under the desk to use AMDS at two o'clock in the morning, which was when the problem tended to occur.

However, I finally managed to catch what was happening with AMDS early on Sunday morning, and decoded the file serialization lock that was pingponging back and forth between nodes in the cluster, and discovered the massive directory.

We fixed the cluster hangs by limiting the application that uses the directory to one node of the cluster for the time being. Cleaning up a directory of this size is a rather slow process when you can't just rename it out of the way and create an empty one.

The application team is having great fun being roughly educated as to the dangers of large directories with files all starting with the same half dozen characters.

Having an architecture oversight group that understood the OpenVMS file system while this application was being written would have prevented this problem from occuring.