Skip to content

KVM: SharedMountPoint HA heartbeat fails when pool path is a subdirectory of a mounted cluster FS ("Mount point is not a mounted filesystem") #14326

Open
#14331
Open
KVM: SharedMountPoint HA heartbeat fails when pool path is a subdirectory of a mounted cluster FS ("Mount point is not a mounted filesystem")#14326
#14331
Labels
Severity:Majorcomponent:kvmcomponent:primary-storagetype:bug
Milestone
4.22.2

Description

@danivogel90
danivogel90
opened on Oct 7, 2026

ISSUE TYPE

Bug Report

COMPONENT NAME

KVM agent / HA (scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh)

CLOUDSTACK VERSION

4.22.1.0, 4.23.0.0 (introduced by #12773)

CONFIGURATION

KVM, Ubuntu 24.04, SharedMountPoint primary storage on IBM Storage Scale (GPFS).
GPFS is mounted at /data01, the primary storage pool path is /data01/vol01.

SUMMARY

#12773 added HA heartbeats for SharedMountPoint pools. The new script
kvmsmpheartbeat.sh requires the pool path itself to be a mount point:

if ! mountpoint -q "$MountPoint"; then
  echo "Mount point is not a mounted filesystem: $MountPoint" >&2
  exit 1

According to the docs, SharedMountPoint is "a file system path local to each
server". A subdirectory of a clustered filesystem (GPFS, OCFS2, GFS2, CephFS ...)
is a valid and common setup. For such a pool the heartbeat write never succeeds:

WARN [kvm.resource.KVMHAMonitor] Write heartbeat for pool [27454846-...] failed:
Mount point is not a mounted filesystem: /data01/vol01; try: 5 of 5.

IMPACT

STEPS TO REPRODUCE

  1. Mount a shared FS at /data01. Create a SharedMountPoint pool with path /data01/vol01.
  2. Upgrade or install the agent to >= 4.22.1.
  3. Agent log shows the warning above every minute; KVMHA/ is never created.

EXPECTED RESULTS

The heartbeat is written to /data01/vol01/KVMHA/hb-<ip>.

SUGGESTED FIX

⚠️ Incomplete – do not apply alone. See the comment below and PR #14331.

Keep the safety intent (do not write the heartbeat to the local root FS when the shared
FS is not mounted). Check the filesystem containing the path instead of requiring the
path itself to be a mount point:

if command -v findmnt >/dev/null 2>&1; then
  mnt_target=$(findmnt -n -o TARGET -T "$MountPoint" 2>/dev/null)
  if [ -z "$mnt_target" ] || [ "$mnt_target" = "/" ]; then
    echo "Mount point is not on a mounted (shared) filesystem: $MountPoint" >&2
    exit 1
  fi
fi

Workaround for now: bind-mount the directory onto itself (mount --bind /data01/vol01 /data01/vol01).

Related: #12773 (SMP heartbeat), #13090 (configurable fence action), #10111 (reboot on heartbeat timeout disabled by default).
Broader context: see enhancement issue #14327 (first-class support for parallel/clustered filesystems).

Activity

  1. weizhouapache commented on Oct 7, 2026

    @weizhouapache
    weizhouapache
    on Oct 7, 2026
    Member

    @danivogel90
    thanks for reporting, it looks like a valid bug

    could you verify the fix and create a pull request if it works ? thanks

  2. danivogel90 commented on Oct 7, 2026

    @danivogel90
    danivogel90
    on Oct 7, 2026
    Author

    Thanks for the quick response! I tested a fix and found two more things that matter for the PR:

    1. The suggested fix in the issue description alone is dangerous.
    Further down, kvmsmpheartbeat.sh has a second check:

    if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
then
   :
else
   if [ "$rflag" == "0" ]
   then
     deleteVMs $MountPoint     # kill -9 on every qemu process using this path
   fi
fi

    Today this is never reached for a subdirectory pool, because the mountpoint -q check exits earlier.
    If only the mountpoint -q check is relaxed, every heartbeat write would kill -9 all VMs on the pool.
    So both checks have to use the same logic. My patch adds a helper is_on_mounted_fs()
    (findmnt -n -o TARGET -T, fallback df -P). It returns true if the path is on a mounted filesystem
    other than /, and it replaces both checks.

    2. check_hbLog reports a live host as DEAD when diff == 0.
    diff=expr $now - $hb`` is followed by if [ $? -ne 0 ], but `expr` exits with status 1 when the result is 0.
    A read in the same second as the write therefore returns `999997` → "Considering host as DEAD".
    Reproduced with the unmodified 4.23.0.0 script: write, then read immediately with `-r -t 60` → DEAD.
    The patch validates `$hb` as numeric and uses `$((now - hb))`.

    Tests (copy of the script with kill -9 disarmed, then live in the agent):

    Case Expected Result
    original, subdirectory of GPFS mount fails exit 1
    patched, subdirectory of GPFS mount, write + read ALIVE, no deleteVMs OK
    patched, real mount point (tmpfs) ALIVE OK
    patched, NFS pool mount point (read) ALIVE OK
    patched, path on root FS refuse, nothing written exit 1, dir empty
    patched, path does not exist refuse exit 1
    original, read right after write (diff=0) ALIVE DEAD
    patched, read right after write / after 3 s, -t 60 ALIVE OK / OK
    patched, after 3 s with -t 1 / corrupt hb file DEAD OK / OK

    The patched script runs live in the KVM agent on one host since 2026-10-07 (Ubuntu 24.04, 4.23.0.0,
    IBM Storage Scale 5.2.3-5, pool /data01/vol01). No heartbeat errors since then. I will open a PR (against 4.22, Fixes #14326) next.

    Update – HA failover test (2026-10-07): 3 KVM hosts, HA-enabled VM on the SharedMountPoint pool on host A,
    host A hard-reset via sysrq (agent autostart disabled so it stays away). The neighbour running the patched script
    reported "Heart is not beating" for both pools (NFS + SharedMountPoint) about 85 s after the reset. The host was marked
    Down and the VM was restarted on another host about 110 s after the reset. No deleteVMs side effects.
    Side observation: while the clustered FS was recovering the failed node, heartbeat writes on the surviving hosts hit
    the script timeout once. The killed write leaves an empty hb-<ip>.<pid> temp file behind. I can add a trap cleanup
    to the PR if you like.

  3. danivogel90
    added a commit that references this issue on Oct 7, 2026

    kvm: fix SharedMountPoint heartbeat when pool path is a subdirectory …

    d62ce35

  4. danivogel90 commented on Oct 7, 2026

    @danivogel90
    danivogel90
    on Oct 7, 2026
    Author

    Thanks @weizhouapache! I verified the fix and opened PR #14331 against 4.22.

    While testing I found two more things, both fixed in the PR:

    1. Relaxing only the mountpoint -q check (as suggested in the description above) is
      dangerous: the later /proc/mounts check would then call deleteVMs, which
      kill -9s all VMs on the pool, on every heartbeat write. The PR uses the same
      check in both places.
    2. check_hbLog reports a live host as DEAD if the heartbeat is read in the same
      second it was written (expr exits 1 when the result is 0).

    Tested with script tests, live in the agent and with an HA failover test (host hard-reset,
    HA VM restarted on another host after ~110 s). Details are in the PR.

  5. github-actions
    added
    component:kvm
    component:primary-storage
    Severity:Major
    type:bug
    on Oct 7, 2026

  6. github-actions commented on Oct 7, 2026

    @github-actions
    github-actionsBot
    on Oct 7, 2026 – with GitHub Actions

    🎯 Triage report

    Confirmed bug: kvmsmpheartbeat.sh (added by #12773) requires the SharedMountPoint pool path itself to be a mount point, but subdirectories of clustered/parallel filesystems (GPFS, OCFS2, GFS2, CephFS) are a valid and documented configuration. This breaks VM-HA heartbeat writes and, combined with reboot.host.and.alert.management.on.heartbeat.timeout=true, can trigger simultaneous hard resets across a whole cluster. A maintainer (@weizhouapache) already confirmed it as a valid bug, and the author has tested a fix and opened PR #14331.

    📊 Assessment

    Dimension Value Reasoning
    Type type:bug Regression in HA heartbeat logic introduced by #12773, confirmed by a maintainer.
    Component component:kvm, component:primary-storage Affects KVM agent HA heartbeat script for SharedMountPoint primary storage.
    Severity Severity:Major Can cause cluster-wide, simultaneous host hard-resets and storage quorum loss in hyperconverged setups; workaround exists (bind-mount).
    Labels type:bug, component:kvm, component:primary-storage, Severity:Major See above
    Coding agent Not suitable Fix already authored, tested, and submitted as PR #14331 by the reporter; no further automated work needed here.

    🔗 Similar issues

    💡 Notes and suggestions
    • PR kvm: fix SharedMountPoint heartbeat when pool path is a subdirectory of a mount #14331 reportedly also fixes a related issue in check_hbLog: expr exits with status 1 when the computed diff is 0, causing a live host to be misreported as DEAD if read happens in the same second as a write.
    • The author's testing (per the comments) covered: subdirectory-of-GPFS-mount heartbeats, real mount points (tmpfs), NFS pools, refusal on root-FS paths and nonexistent paths, and a live 3-host HA failover test with sysrq hard reset — all passing with the patch.
    • One side issue noted during failover testing: a killed heartbeat write can leave behind an empty hb-<ip>.<pid> temp file; the author offered to add a trap cleanup to the PR — maintainers may want to confirm this is included before merge.
    • Recommend maintainers prioritize review of PR kvm: fix SharedMountPoint heartbeat when pool path is a subdirectory of a mount #14331 given the severity (cluster-wide reboot risk) and that it's already tested.

    Generated by Daily Issue Triage · sonnet50 92.1K · ◷

    Add this agentic workflows to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9

  7. danivogel90 commented on Oct 7, 2026

    @danivogel90
    danivogel90
    on Oct 7, 2026
    Author

    FYI: the trap cleanup for the temporary hb-<ip>.<pid> file mentioned in the issue is already included in d62ce35.

  8. DaanHoogland
    added this to the 4.22.2 milestone on Oct 19, 2026
  9. DaanHoogland
    linked a pull request that will close this issuekvm: fix SharedMountPoint heartbeat when pool path is a subdirectory of a mount #14331on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Severity:Majorcomponent:kvmcomponent:primary-storagetype:bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions