Full File-only Cache with hard N-day eviction

A file cannot be stored on HDFS for more than 30 days due to GDPR requirements. Therefore, every 25 days we purge old files to meet compliance requirements. When we bring a file into a cache, we can no longer store it for longer than 5 days. Otherwise, we may run into compliance violations, as a file can be 25 days old when we bring it to GPFS caching.

We use our Hydrocache solution with AFM and HDFS to serve files for ML training. Once a file is read by the GPU cluster, the whole file needs to be read otherwise the training run needs to be aborted and GPU hours wasted. Therefore, we need either the whole file to be read from cache or no file at all. A partial cache read is not viable because the file stored at HDFS might have been replaced with a new version by our purge process.

We are looking at a cache strategy that looks like as follows:

0. If a file is not fully cached, we don't serve it from GPFS caching. A client should be redirected to read from HDFS.
1. When a file open from GPFS succeeds and a reader starts reading a file from the cache, it never falls back to HDFS. When reading from GPFS fails after retry, throw an error -> fail the job -> start the new job reads from HDFS.
2. Cached copy should not be invalidated/changed when there are active readers.
3. When we reach 90% of fileset capacity, stop ingesting/caching more files.
4. Cache Eviction & Compliance Policy:
a. Each cached file will be cached only for 5 days.
b. Block new reads on the 5th day.
c. Evict cached files on the 6th day.

To do this we need enhanced functionality:

1. NN Proxy rejects open() when either:
a. A file is not fully cached
b. Or it has been more than N days since it was fully cached. Basically, we should block new opens once a file has been cached for more than N days.
i. add a fully-cached timestamp as a separate property for each cached file, which gives O(1) when deciding whether we need to reject new Open() on the (N+1)th day for a cached file (TTL of N+1 days). This timestamp should be updated each time a file is fully cached. This timestamp serves as SoT fully cached time for a file.
2. Record the timestamp T when a file is fully cached.
a. Append these timestamps with the full file path to a hourly bucketed file. We will build an eviction service, to scan these files, to evict cached files older than 5 days. Before we evict a file, we check the SoT and evict if it is indeed older than 5 days.