Support online trim / discard of NSD freed/released blocks to optimise NVME backend performances

We are observing important degradation of GPFS read/write performances that appears to be related to the lack of online trim/discard actions for freed/released blocks on the NSDs.

Performances go back to normal once mmreclaimspace is executed and then go back to a bad state periodically depending on the amount of data stored / deleted on the FS.

mmreclaimspace does not seem to be the right answer to that problem because when it runs :

- it has to scan all the FS and perform a lot of trim/discard actions based on the specified threshold in a single run

- the impact of the mmreclaimspace running concurrently with production workloads can be really high because of that (there is no real QOS support for it)

- reducing the threshold reduces the amount of discarded blocks and just never "fix" some of them, still letting the underlying NVME having to deal with garbage collection at write time later

- it is hard / almost impossible to detect when / how often to run it as it depends of the workload write ratio that can fluctuate and the way the blocks are used across the NSDs

Having a sort of online block trim/discard logic (probably batching discard operations on a short time window (few/dozens seconds)) when releasing blocks on NSD of the NVME type would be in our opinion a better way to reach consistent performances and avoid large period of bad performances due to underlying garbage collection cycles with Spectrum Scale on NVME based storage.

Having such an online trim/discard logic has a per FS option, with a few knobs to tune the behaviour (batching, internal threshold, ...)  and some metrics to see what is performed in the background would help to tune GPFS to really leverage the capabilities of the underlying storage and have consistent and good performances over time.