Extend readReplicaPolicy and "failure group" to nodes, to help localize reads in a stretched cluster.

With a 2-site "stretched cluster", readReplicaPolicy can be used to choose which replica is used for reading, except... our options don't really handle complex network topologies. The "fastest" setting is often not a good choice because network latency is still comparable to HDD seek times.

This RFE proposes that just as disks can be tagged with a failure group, that nodes also be able to be tagged with failure groups. Then also add a new setting for readReplicaPolicy, perhaps "failureGroup", to cause a replica with the same failure group tag as the node to be preferred for reading. (If not available, perhaps fall back to "local".)

This function would extend to multi-cluster, so the failure group of a node in a consuming cluster would prefer the replica with the same failure group in the owning (storage) cluster. This would require some cross-cluster coordination, but there are already other things that require this (for example uid/gid, or the fabric number in verbsPorts).

Potentially the failure group attribute could also be used with non-replicated writes to a file system, to prefer writing locally (that could allow replication to be deferred until a later mmapplypolicy, sparing applications from some write latency at the cost of possibly losing recent work).

Potentially the failure group attribute could also be used to optimize commands that need to scan metadata and the file system, such as mmrestripefs and mmapplypolicy (see also RFE 149675).

Potentially some other attribute, "site", could be added to both disks and nodes - if a site had more than one disk failure group, and we had sufficient replication, this would be useful. But we don't typically see a replication factor of 3, and failure group already seems to capture the important cases, it seems simplest to continue to overload the meaning of "failure group".

Potentially an AFM fileset could have the afmGateway parameter extended to use a failure group for the preferred gateway node - this could optimize network access between application nodes and the gateway (presumably the application is primarily using nodes in the same "failure group" as specified in the fileset).

To really fill out this feature, include system node classes to encompass nodes in a particular failure group.