Repository navigation
Deleted KVM host is automatically re-added to the cluster after reboot #13693
Description
problem
We removed a host from a CloudStack cluster containing multiple KVM hosts. Before deletion, the host was placed into maintenance mode, as required by CloudStack in order to remove it safely.
After the host had been deleted from CloudStack, the underlying physical server was rebooted because updates had been applied. The reboot itself was not expected to have any effect on CloudStack, since the host had already been removed from the management plane.
Unexpectedly, after the reboot, CloudStack rediscovered the machine and added it back to the same cluster as a new host. We also verified in the CloudStack database that the host entry had first been deleted and then recreated as a new record.
This behavior suggests that CloudStack is still able to rediscover a previously removed KVM host when the machine comes back online, even though it had already been deleted from CloudStack.
Expected behavior:
Once a host has been removed from CloudStack, rebooting the underlying physical server should not cause the host to be automatically rediscovered and re-added to the same cluster.
Actual behavior:
The deleted host was automatically added back to the cluster as a new host after the physical server reboot.
Environment:
CloudStack: 4.22.1.0
Hypervisor: KVM
Cluster: multiple hosts
Host state before deletion: maintenance mode
Host status after deletion: removed from CloudStack
Database observation: host entry was deleted and later recreated as a new entry
Additional details:
We did not manually re-add the host after deletion.
The host appears to have been rediscovered automatically by CloudStack after the reboot.
The issue occurred in a multi-host cluster environment.
versions
CloudStack version: 4.22.1.0
OS and database versions:
We observed this behavior in two CloudStack installations:
Management Server OS: Ubuntu 24.04 LTS
Database: MySQL 8.x
KVM Host OS: AlmaLinux 9.8 and Ubuntu 24.04 LTS
The steps to reproduce the bug
-
Migrate all VMs off the host.
-
Put the host into maintenance mode.
-
Remove the host from the cluster in CloudStack.
-
Reboot the physical server.
What to do about it?
No response
Activity
boring-cyborg commentedon Jul 24, 2026
Thanks for opening your first issue here! Be sure to follow the issue template!
DaanHoogland commentedon Jul 24, 2026
@wido , you guys use the autodiscovery of hosts right? Can you comment?
@Eddi1306 , I think you need to make sure your remove the agent from the host to prevent the re-adding it?
sbrueseke commentedon Jul 24, 2026
@DaanHoogland yes, you are correct, but I would expect that after deleting a host in ACS the host would not be able to rejoin automatically only by reboot. In the DB the original host's state will change to removed and after a simple reboot a new host with a new id will be added to the DB.
So maybe a better workflow is that when you delete a host the agent config will also be deleted from the host.
wido commentedon Jul 27, 2026
I think you have a valid point here. If the host was removed from the UI, it should not re-add itself. Let me check on this.
weizhouapache commentedon Jul 27, 2026
You can try setting the global configuration add.host.on.service.restart.kvm to false.
This option was introduced in #4228
wido commentedon Jul 27, 2026
@weizhouapache I don't fully agree with you. I made this PR to address it: #13719
We remove the GUID in the 'hosts' table and therefor can't find the host afterwards anymore. By preserving it we can.