Please see The HDF Group's new Support Portal for the latest information.
The parallel version of the HDF5 metadata cache has long supported the maintenance of a "minimum clean fractions" -- which means that the metadata cache will flush entries as necessary to keep a specified fraction of the cache clean at all times.
1.8.3, we extended this feature to the serial version of the metadata cache to deal with the performance issues encountered when the cache filled up with dirty metadata, and then flushed every entry in the cache before evicting one of them to make space for a new entry -- colloquially known as the "metadata blizzard" bug.
The slowdown observed in the "create a lot of datasets" test between 1.8.2 and the 1.8.3 release candidate was caused by an interaction between this change and the adaptive cache resize code.
Specifically, the adaptive cache resize code monitors the metadata cache hit rate, and increases the size of the cache if the hit rate drops below some threshold. However, if the min clean fraction is a significant fraction of the cache size, it is possible that the coldest min clean fraction of the entries will be hot enough to generate a lot of metadata writes -- to a first order approximation, this is what caused the slowdown in the "create a lot of datasets" test.
For the serial case, the problem is relatively easy to solve -- just keep the min clean fraction very small, say 0.01, which avoids the problem and still deals with the metadata blizzard bug.
The problem is more difficult in the parallel case, as there a larger minimum clean size is needed to allow the cache to make space for new entries without flushing entries during periods when metadata writes are forbidden.
For the 1.8.3 release, we are dealing with the problem by modifying the default metadata cache configuration in the serial case to set the minimum clean fraction to 0.01 (along with some other minor tweaks that we think will help performance). For now, we will leave the default metadata cache configuration in the parallel case essentially unchanged.
In the longer term, it is clear that we have some work to do on the
adaptive metadata cache resize code -- hopefully, that will show up in
1.8.4. For those who see this problem in the parallel case before
then, the suggested workaround is to take manual control of the
metadata cache, and set its size to be ~1.5 times the expected working
set size while keeping the minimum clean fraction at 0.3.
- - Last modified: 13 February 2014