Repository navigation
Ensure compliance with storage access groups when mounting disabled storage pools - #13990
Conversation
…torage pools - connect to only storage pools with empty tags when access group is empty
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 4.22 #13990 +/- ##
============================================
- Coverage 17.79% 17.78% -0.01%
+ Complexity 15995 15989 -6
============================================
Files 5928 5928
Lines 534306 534328 +22
Branches 65383 65386 +3
============================================
- Hits 95069 95033 -36
- Misses 428467 428525 +58
Partials 10770 10770
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18998 |
kiranchavala
left a comment
There was a problem hiding this comment.
@sureshanaparti getting the following exception
(localcloud) :cat: > configure storageaccess storageid=809eff3f-acf8-3a3f-8a18-c8630da5608e storageaccessgroups=group1
2026-09-29 06:08:00,888 DEBUG [c.c.s.StorageManagerImpl] (API-Job-Executor-31:[ctx-87f44cdf, job-37, ctx-9a23c72f]) (logid:ca2e6340) Updating Storage Pool Access Group Maps to :[group1]
2026-09-29 06:08:00,910 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-31:[ctx-87f44cdf, job-37, ctx-9a23c72f]) (logid:ca2e6340) Conflict detected: Hosts using the storage pool that need to be disconnected or connected to the pool: Host IDs and volumes: {Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"}=[Volume {"id":1,"instanceId":1,"name":"ROOT-1","uuid":"d36faa6f-4ddc-4b9c-ac6d-ec903c4b6719","volumeType":"ROOT"}]}
2026-09-29 06:08:00,922 DEBUG [o.a.c.a.c.a.s.ConfigureStorageAccessCmd] (API-Job-Executor-31:[ctx-87f44cdf, job-37, ctx-9a23c72f]) (logid:ca2e6340) Failed to configure storage access com.cloud.utils.exception.CloudRuntimeException: Storage access groups cannot be updated as they are currently in use by some hosts. Please check the logs.
@kiranchavala this seems to be expected when there are few hosts using the pool and marked to disconnect during storage access configuration. |
my mistake @sureshanaparti there were some systemvm running on the storage |
kiranchavala
left a comment
There was a problem hiding this comment.
LGTM Tested manually
Step 1: Assign storage access groups
cmk configure storageaccess storageid=<PS1> storageaccessgroups=group1 # PS1 = group1
# PS2 = no SAG (leave as is)
cmk configure storageaccess hostid=<HostA> storageaccessgroups=group1
Step 2: Disable both storages
cmk update storagepool id=<PS1> enabled=false
cmk update storagepool id=<PS2> enabled=false
Step 3: Config OFF
cmk update configuration name=mount.disabled.storage.pool value=false clusterid=<cluster>
cmk reconnect host id=<HostA>
cmk reconnect host id=<HostB>
Expected: neither host mounted PS1 or PS2.
Step 4: Config ON, host with a matching SAG
cmk update configuration name=mount.disabled.storage.pool value=true clusterid=<cluster>
cmk reconnect host id=<HostA>
Expected: HostA mounts PS1 and PS2. PS1 matches group1, and PS2 has no SAG.
Step 5: Config ON, host with no SAG (tests the fix)
cmk reconnect host id=<HostB>
Expected: HostB mounts PS2 only. Without the fix, HostB would also mount PS1.
Step 6: Config ON, host with a non-matching SAG (tests the fix)
cmk configure storageaccess hostid=<HostA> storageaccessgroups=group2
cmk reconnect host id=<HostA>
Expected: HostA mounts PS2 only, and PS1 is disconnected from it. Without the fix HostA would keep PS1.
Step 7: Real reboot / agent restart
On HostA and HostB:
systemctl restart cloudstack-agent # or reboot the host
Expected: the same result as Steps 5 and 6.
LOGS
[root@ref-trl-12434-k-Mol8-kiran-chavala-mgmt1 ~]# grep -E "connecting host to shared pool|Removing pool|Connection removed between storage pool" $LOG
2026-09-30 11:52:17,650 DEBUG [c.c.s.StorageManagerImpl] (API-Job-Executor-37:[ctx-8a5651d7, job-49, ctx-7115a784]) (logid:d5d9dc43) Removing pool StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} from host Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"}
2026-09-30 11:52:17,693 INFO [o.a.c.s.d.p.DefaultHostListener] (API-Job-Executor-37:[ctx-8a5651d7, job-49, ctx-7115a784]) (logid:d5d9dc43) Connection removed between storage pool: StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} and host: Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"}
2026-09-30 11:52:17,695 DEBUG [c.c.s.StorageManagerImpl] (API-Job-Executor-37:[ctx-8a5651d7, job-49, ctx-7115a784]) (logid:d5d9dc43) Removing pool StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} from host Host {"id":2,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm2","type":"Routing","uuid":"44362db9-8fc1-49e9-8d4e-fa75b2205e8e"}
2026-09-30 11:52:17,732 INFO [o.a.c.s.d.p.DefaultHostListener] (API-Job-Executor-37:[ctx-8a5651d7, job-49, ctx-7115a784]) (logid:d5d9dc43) Connection removed between storage pool: StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} and host: Host {"id":2,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm2","type":"Routing","uuid":"44362db9-8fc1-49e9-8d4e-fa75b2205e8e"}
2026-09-30 12:02:00,652 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-7:[ctx-98b66346]) (logid:107d551e) Host Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"} connected, connecting host to shared pool StoragePool {"id":2,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri2","poolType":"NetworkFilesystem","uuid":"fd5d4a9d-811d-3ffd-90fc-de05156e01e0"} and sending storage pool information ...
2026-09-30 12:02:00,700 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-7:[ctx-98b66346]) (logid:107d551e) Removing pool StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} from host Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"}
2026-09-30 12:02:00,838 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-7:[ctx-98b66346]) (logid:107d551e) Connection removed between storage pool: StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} and host: Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"}
2026-09-30 12:04:03,646 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-8:[ctx-eca5efa1]) (logid:a5c52fe0) Host Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"} connected, connecting host to shared pool StoragePool {"id":2,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri2","poolType":"NetworkFilesystem","uuid":"fd5d4a9d-811d-3ffd-90fc-de05156e01e0"} and sending storage pool information ...
2026-09-30 12:04:03,689 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-8:[ctx-eca5efa1]) (logid:a5c52fe0) Host Host {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"59e48ea4-292f-40e5-bdd6-f81f2c90bb41"} connected, connecting host to shared pool StoragePool {"id":1,"name":"ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","poolType":"NetworkFilesystem","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e"} and sending storage pool information ...
[root@ref-trl-12434-k-Mol8-kiran-chavala-kvm1 ~]# grep -E "ModifyStoragePoolCommand|DeleteStoragePoolCommand" $LOG | grep -E "$POOL1|$POOL2" | tail -20
2026-09-30 11:52:17,656 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:d5d9dc43) Request:Seq 1-3016285850431411833: { Cmd , MgmtId: 32986858783620, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"1","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e","host":"10.0.32.4","path":"/acs/primary/ref-trl-12434-k-Mol8-kiran-chavala/ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","type":"NetworkFilesystem"},"_localPath":"/mnt//809eff3f-acf8-3a3f-8a18-c8630da5608e","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-30 11:52:17,657 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:d5d9dc43) Processing command: com.cloud.agent.api.DeleteStoragePoolCommand
2026-09-30 11:59:20,752 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:d9847904) Request:Seq 1-3016285850431411843: { Cmd , MgmtId: 32986858783620, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"1","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e","host":"10.0.32.4","path":"/acs/primary/ref-trl-12434-k-Mol8-kiran-chavala/ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//809eff3f-acf8-3a3f-8a18-c8630da5608e","details":{},"wait":"300","bypassHostMaintenance":"false"}}] }
2026-09-30 11:59:20,752 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:d9847904) Processing command: com.cloud.agent.api.ModifyStoragePoolCommand
2026-09-30 12:02:00,662 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:107d551e) Request:Seq 1-4810688826961821698: { Cmd , MgmtId: 32986858783620, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"2","uuid":"fd5d4a9d-811d-3ffd-90fc-de05156e01e0","host":"10.0.32.4","path":"/acs/primary/ref-trl-12434-k-Mol8-kiran-chavala/ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri2","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//fd5d4a9d-811d-3ffd-90fc-de05156e01e0","details":{},"wait":"300","bypassHostMaintenance":"false"}}] }
2026-09-30 12:02:00,662 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:107d551e) Processing command: com.cloud.agent.api.ModifyStoragePoolCommand
2026-09-30 12:02:00,759 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:107d551e) Request:Seq 1-4810688826961821699: { Cmd , MgmtId: 32986858783620, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"1","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e","host":"10.0.32.4","path":"/acs/primary/ref-trl-12434-k-Mol8-kiran-chavala/ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","type":"NetworkFilesystem"},"_localPath":"/mnt//809eff3f-acf8-3a3f-8a18-c8630da5608e","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-30 12:02:00,759 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:107d551e) Processing command: com.cloud.agent.api.DeleteStoragePoolCommand
2026-09-30 12:04:03,656 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:a5c52fe0) Request:Seq 1-1148417904979476482: { Cmd , MgmtId: 32986858783620, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"2","uuid":"fd5d4a9d-811d-3ffd-90fc-de05156e01e0","host":"10.0.32.4","path":"/acs/primary/ref-trl-12434-k-Mol8-kiran-chavala/ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri2","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//fd5d4a9d-811d-3ffd-90fc-de05156e01e0","details":{},"wait":"300","bypassHostMaintenance":"false"}}] }
2026-09-30 12:04:03,657 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:a5c52fe0) Processing command: com.cloud.agent.api.ModifyStoragePoolCommand
2026-09-30 12:04:03,739 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:a5c52fe0) Request:Seq 1-1148417904979476483: { Cmd , MgmtId: 32986858783620, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"1","uuid":"809eff3f-acf8-3a3f-8a18-c8630da5608e","host":"10.0.32.4","path":"/acs/primary/ref-trl-12434-k-Mol8-kiran-chavala/ref-trl-12434-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//809eff3f-acf8-3a3f-8a18-c8630da5608e","details":{},"wait":"300","bypassHostMaintenance":"false"}}] }
2026-09-30 12:04:03,739 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:a5c52fe0) Processing command: com.cloud.agent.api.ModifyStoragePoolCommand
[root@ref-trl-12434-k-Mol8-kiran-chavala-kvm1 ~]#
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19358 |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17054)
|
Description
The PR addresses the following issue:
During host connect process, management server chooses which storage pool to connect to, using storage pool scope (CLUSTER or ZONE) and storage access group (tags). In the scenario where configuration
mount.disabled.storage.poolis ON, the behaviour is different/buggy.The above edge case with the below changes are handled when the config
mount.disabled.storage.poolis ON:Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Scenarios:
mount.disabled.storage.pool is OFF:
mount.disabled.storage.pool is ON and storage access group of host is "group1":
mount.disabled.storage.poolwas OFF. Additionally, the host would connect to all storage pools in Disabled state with the below filters:mount.disabled.storage.pool is ON and storage access group of host is empty:
mount.disabled.storage.poolwas OFF. Additionally, the host would connect to all storage pools in Disabled state with the below filters:How did you try to break this feature and the system with this change?