Reliability option for when suspension of disk occurs on disk outages

IBM: Currently when allocating disk space, gpfs only consider the disk status(ready,suspended/to-be-emptied, emptied, replacing, replacement) but not "availability"(up, down, recovering, unrecovered).

for "availability", it will impact whether GPFS will read/write data into the NSD("up" means GPFS can read/write; "down" means GPFS can't read/write and need to try other replica).
for "status", if it's "suspended/to-be-emptied", that indicates that,the data on disk will be restriped out and thus GPFS won't allocate new blocks on these disks to reduce the to-be-restriped-out data size.

there are several reasons for GPFS to mark the disk as down: physical cable loose, physical disk error, NSD server is down or network exception etc. GPFS can't tell which reason causes the EIO and can't say how long the disk will be back. So, GPFS won't consider disk "status" when allocating new blocks. this is one kind of tradeoff: if we don't allocate new blocks on down disks, if the disk will be coming back soon, then GPFS has to consider to re-balance disk space. if the disk can't be back for long time, GPFS should not allocate new space on the down disks.

so, that's why auto recovery is introduced to mark down disk as susppended if the disk is still down after waiting for threshold of time.

Request: the above makes sense when you have locality rules and you want to ensure that data doesn't get allocated to the wrong server forcing the need for a restripe. It allows for a period of time where a node or disk can go out of service and blocks will still be allocated to it. Especially important below DB2 and similar systems.

But in a general FPO file system the loss of a node or disk is not that important. What is more important is maintaining the reliability of the data and retaining three copies of the blocks. As clusters get larger the chances of additional disk outages rises and if two more disks go out that have a block on them that is the same as the initial drive that file is now corrupt.

So for this type of cluster I would like a file system configuration that suspends a disk immediately GPFS recognizes that it is down rather than when the disk wait time for recovery runs out. This will force new blocks to be written to other disks and/or nodes in the cluster maintaining replication count but at the possible expense of loss of locality. On the other hand it also reduces the number of blocks that must be replicated when the automatic restripe is started reducing the load on the cluster and speeding the recovery.

That's a trade off that to me is very acceptable. Locality I can repair.