Limitation in the connmgr pod in OpenShift

We discussed a limitation in the connmgr pod in OpenShift with Joseph Dain in Hamburg. Due to the cgroup settings in OpenShift a pod can only have 1024 pids. Since every scan creates ~100 or more threads we reach this threshold soon causing the whole connmgr pod to become unreliable. A quick solution that we came up with Joseph is to increase the number of connmgr pods to distribute the scans to several pods and several nodes.

There still seems to be an issue with the synchronization if we have more than one connmgr pod but at least the scanned data gets ingested into the DB.

We also discussed an improved architecture. Our proposal is to have kind of an orchestrator in OpenShift that:

• Every time a scan of a qtree is initiated the orchestrator:

  • creates a new pod

  • mounts the qtree in the pod so the pod doesn't have to mount it himself. This could possible remove the necessity for some high privileges in the SCC in OpenShift

  • Once the pod finishes the scan of a qtree the result it put into the DB and the pod is removed by the orchestrator

  • The orchestrator keeps track of the status of each scan

  • Another problem that can be solved with this approach is the possible exhaustion of file handles:

    • In the current connmgr pod every qtree stays mounted and if a new data connection can not be added the connmgr pod creates a directory under /nfs-scanner/ anyhow. Given more than 30000 qtrees and possibly several failed attempts to create a new data connection we may easily have a huge number of mounts and emtpy/useless directories under /nfs-scanner