Enhancement: first-class support for parallel/clustered filesystems (IBM Storage Scale, WEKA, Quobyte, ...) as KVM primary storage

[danivogel90]

ISSUE TYPE

Enhancement Request

COMPONENT NAME

KVM agent, Primary storage (SharedMountPoint), HA/fencing, Documentation

CLOUDSTACK VERSION

4.22.x / 4.23.x and later

SUMMARY

SharedMountPoint is the natural (and often the only) way to run KVM primary storage on
parallel filesystems such as IBM Storage Scale (GPFS), WEKA, Quobyte, and also
on GFS2/OCFS2/CephFS. These setups are increasingly common, often hyperconverged, but
they seem to be rarely covered by tests. Assumptions that only hold for NFS or local
storage keep slipping in, for example:

• the pool path must be its own mount point (see KVM: SharedMountPoint HA heartbeat fails when pool path is a subdirectory of a mounted cluster FS ("Mount point is not a mounted filesystem") #14326),
• heartbeat or I/O hiccups lead to an immediate sysrq hard reset. On hyperconverged
  parallel-FS clusters a simultaneous fence of several hosts destroys the storage quorum
  (see Disabled the setting reboot.host.and.alert.management.on.heartbeat.timeout by default #10111, KVM: make storage heartbeat fence action configurable (graceful-reboot / restart-agent / log-only) #13090),
• NFS-specific handling (e.g. stale-handle/remount logic) does not apply to these
  filesystems.

Suggestions:

• treat SharedMountPoint on a parallel FS as a documented, tested configuration (at least
  one CI scenario with the pool in a subdirectory of a cluster mount),
• consider an explicit "clustered/parallel filesystem" pool flavour or capability so that
  heartbeat, fencing and mount checks can be adapted per filesystem,
• document recommended HA/fencing settings for hyperconverged deployments.

Concrete example: #14326 (SMP heartbeat requires the pool path to be a mount point).

Happy to help test with IBM Storage Scale in a hyperconverged KVM setup.

Related: #12773 (SMP heartbeat), #13090 (configurable fence action), #10111 (reboot on heartbeat timeout disabled by default).