KVM: SharedMountPoint HA heartbeat fails when pool path is a subdirectory of a mounted cluster FS ("Mount point is not a mounted filesystem")

[danivogel90]

ISSUE TYPE

Bug Report

COMPONENT NAME

KVM agent / HA (scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh)

CLOUDSTACK VERSION

4.22.1.0, 4.23.0.0 (introduced by #12773)

CONFIGURATION

KVM, Ubuntu 24.04, SharedMountPoint primary storage on IBM Storage Scale (GPFS).
GPFS is mounted at /data01, the primary storage pool path is /data01/vol01.

SUMMARY

#12773 added HA heartbeats for SharedMountPoint pools. The new script
kvmsmpheartbeat.sh requires the pool path itself to be a mount point:

if ! mountpoint -q "$MountPoint"; then
  echo "Mount point is not a mounted filesystem: $MountPoint" >&2
  exit 1

According to the docs, SharedMountPoint is "a file system path local to each
server". A subdirectory of a clustered filesystem (GPFS, OCFS2, GFS2, CephFS ...)
is a valid and common setup. For such a pool the heartbeat write never succeeds:

WARN [kvm.resource.KVMHAMonitor] Write heartbeat for pool [27454846-...] failed:
Mount point is not a mounted filesystem: /data01/vol01; try: 5 of 5.

IMPACT

• VM-HA cannot use the heartbeat for this pool (the host looks dead to the investigator).
• With reboot.host.and.alert.management.on.heartbeat.timeout=true (default before 4.23,
  see Disabled the setting reboot.host.and.alert.management.on.heartbeat.timeout by default #10111), every host in the cluster hard-resets via sysrq about 5 minutes after
  the agent upgrade, all at the same time. In hyperconverged setups this also takes down
  the storage cluster quorum.

STEPS TO REPRODUCE

1. Mount a shared FS at /data01. Create a SharedMountPoint pool with path /data01/vol01.
2. Upgrade or install the agent to >= 4.22.1.
3. Agent log shows the warning above every minute; KVMHA/ is never created.

EXPECTED RESULTS

The heartbeat is written to /data01/vol01/KVMHA/hb-<ip>.

SUGGESTED FIX

⚠️ Incomplete – do not apply alone. See the comment below and PR #14331.

Keep the safety intent (do not write the heartbeat to the local root FS when the shared
FS is not mounted). Check the filesystem containing the path instead of requiring the
path itself to be a mount point:

if command -v findmnt >/dev/null 2>&1; then
  mnt_target=$(findmnt -n -o TARGET -T "$MountPoint" 2>/dev/null)
  if [ -z "$mnt_target" ] || [ "$mnt_target" = "/" ]; then
    echo "Mount point is not on a mounted (shared) filesystem: $MountPoint" >&2
    exit 1
  fi
fi

Workaround for now: bind-mount the directory onto itself (mount --bind /data01/vol01 /data01/vol01).

Related: #12773 (SMP heartbeat), #13090 (configurable fence action), #10111 (reboot on heartbeat timeout disabled by default).
Broader context: see enhancement issue #14327 (first-class support for parallel/clustered filesystems).

[weizhouapache]

@danivogel90
thanks for reporting, it looks like a valid bug

could you verify the fix and create a pull request if it works ? thanks

[danivogel90]

Thanks for the quick response! I tested a fix and found two more things that matter for the PR:

1. The suggested fix in the issue description alone is dangerous.
Further down, kvmsmpheartbeat.sh has a second check:

if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
then
   :
else
   if [ "$rflag" == "0" ]
   then
     deleteVMs $MountPoint     # kill -9 on every qemu process using this path
   fi
fi

Today this is never reached for a subdirectory pool, because the mountpoint -q check exits earlier.
If only the mountpoint -q check is relaxed, every heartbeat write would kill -9 all VMs on the pool.
So both checks have to use the same logic. My patch adds a helper is_on_mounted_fs()
(findmnt -n -o TARGET -T, fallback df -P). It returns true if the path is on a mounted filesystem
other than /, and it replaces both checks.

2. check_hbLog reports a live host as DEAD when diff == 0.
diff=expr $now - $hb`` is followed by if [ $? -ne 0 ], but `expr` exits with status 1 when the result is 0.
A read in the same second as the write therefore returns `999997` → "Considering host as DEAD".
Reproduced with the unmodified 4.23.0.0 script: write, then read immediately with `-r -t 60` → DEAD.
The patch validates `$hb` as numeric and uses `$((now - hb))`.

Tests (copy of the script with kill -9 disarmed, then live in the agent):

Case	Expected	Result
original, subdirectory of GPFS mount	fails	exit 1
patched, subdirectory of GPFS mount, write + read	ALIVE, no deleteVMs	OK
patched, real mount point (tmpfs)	ALIVE	OK
patched, NFS pool mount point (read)	ALIVE	OK
patched, path on root FS	refuse, nothing written	exit 1, dir empty
patched, path does not exist	refuse	exit 1
original, read right after write (diff=0)	ALIVE	DEAD
patched, read right after write / after 3 s, -t 60	ALIVE	OK / OK
patched, after 3 s with -t 1 / corrupt hb file	DEAD	OK / OK

The patched script runs live in the KVM agent on one host since 2026-10-07 (Ubuntu 24.04, 4.23.0.0,
IBM Storage Scale 5.2.3-5, pool /data01/vol01). No heartbeat errors since then. I will open a PR (against 4.22, Fixes #14326) next.

Update – HA failover test (2026-10-07): 3 KVM hosts, HA-enabled VM on the SharedMountPoint pool on host A,
host A hard-reset via sysrq (agent autostart disabled so it stays away). The neighbour running the patched script
reported "Heart is not beating" for both pools (NFS + SharedMountPoint) about 85 s after the reset. The host was marked
Down and the VM was restarted on another host about 110 s after the reset. No deleteVMs side effects.
Side observation: while the clustered FS was recovering the failed node, heartbeat writes on the surviving hosts hit
the script timeout once. The killed write leaves an empty hb-<ip>.<pid> temp file behind. I can add a trap cleanup
to the PR if you like.

[danivogel90]

Thanks @weizhouapache! I verified the fix and opened PR #14331 against 4.22.

While testing I found two more things, both fixed in the PR:

1. Relaxing only the mountpoint -q check (as suggested in the description above) is
   dangerous: the later /proc/mounts check would then call deleteVMs, which
   kill -9s all VMs on the pool, on every heartbeat write. The PR uses the same
   check in both places.
2. check_hbLog reports a live host as DEAD if the heartbeat is read in the same
   second it was written (expr exits 1 when the result is 0).

Tested with script tests, live in the agent and with an HA failover test (host hard-reset,
HA VM restarted on another host after ~110 s). Details are in the PR.