Batch resource_count UPDATEs across tags to reduce lock contention by nvazquez · Pull Request #13915 · apache/cloudstack

nvazquez · 2026-08-19T00:57:29Z

Description

Concurrent restoreVirtualMachine on KVM clusters that have tagged storage limits configured (resource.limit.storage.tags) was failing at scale with MySQL "Lock wait timeout exceeded" (errcode 1205). On the worst-affected clusters the failure rate for reimage-vm reached ~43%, and >98% of failures during storm windows traced to RestoreVMCmdByAdmin bottoming out at ResourceCountDaoImpl.updateCountByDeltaForIds.

Root cause: the volume and primary_storage resource-count entry points iterated the configured tag list (the untagged sentinel plus each storage tag) and issued one UPDATE cloud.resource_count per (type, tag) pair. Each UPDATE acquired X-locks on the account row and every parent domain row for that pair, all held until the outer restoreVirtualMachine transaction committed. With multiple sequential UPDATEs per restore and concurrent callers serializing on the shared rows, the in-transaction lock-acquire chain exceeded innodb_lock_wait_timeout (50s default).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

…pache#835) Concurrent restoreVirtualMachine on KVM clusters that have tagged storage limits configured (resource.limit.storage.tags) was failing at scale with MySQL "Lock wait timeout exceeded" (errcode 1205). On the worst-affected clusters the failure rate for reimage-vm reached ~43%, and >98% of failures during storm windows traced to RestoreVMCmdByAdmin bottoming out at ResourceCountDaoImpl.updateCountByDeltaForIds. Root cause: the volume and primary_storage resource-count entry points iterated the configured tag list (the untagged sentinel plus each storage tag) and issued one UPDATE cloud.resource_count per (type, tag) pair. Each UPDATE acquired X-locks on the account row and every parent domain row for that pair, all held until the outer restoreVirtualMachine transaction committed. With multiple sequential UPDATEs per restore and concurrent callers serializing on the shared rows, the in-transaction lock-acquire chain exceeded innodb_lock_wait_timeout (50s default). Fix: add removeResourceReservationIfNeededAndIncrementResourceCountForTags and decrementResourceCountForTags helpers that resolve the union of resource_count row IDs across the full tag list and issue a single batched UPDATE per ResourceType. Migrate all four affected entry points onto these helpers: - incrementVolumeResourceCount - decrementVolumeResourceCount - incrementVolumePrimaryStorageResourceCount - decrementVolumePrimaryStorageResourceCount For an N-tag configuration this collapses 2N sequential UPDATEs to 2 Signed-off-by: Ganesh Maharaj Mahalingam <g.mahalingam@apple.com> Co-authored-by: Ganesh Maharaj Mahalingam <g.mahalingam@apple.com> (cherry picked from commit a8bcb52cee166e362e4ec2744a322ae2588cfe5b)

nvazquez · 2026-08-19T01:00:20Z

@blueorangutan package

blueorangutan · 2026-08-19T01:02:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-08-19T01:06:07Z

Codecov Report

❌ Patch coverage is 9.67742% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.77%. Comparing base (7fc063e) to head (eb97e4b).

Files with missing lines	Patch %	Lines
.../cloud/resourcelimit/ResourceLimitManagerImpl.java	9.67%	54 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #13915      +/-   ##
============================================
- Coverage     17.77%   17.77%   -0.01%     
+ Complexity    15983    15981       -2     
============================================
  Files          5928     5928              
  Lines        534301   534347      +46     
  Branches      65382    65393      +11     
============================================
- Hits          94987    94979       -8     
- Misses       428568   428622      +54     
  Partials      10746    10746

Flag	Coverage Δ
uitests	`3.69% <ø> (ø)`
unittests	`18.85% <9.67%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2026-08-19T01:59:44Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18907

DaanHoogland · 2026-09-01T11:38:02Z

+                // Single batched UPDATE per ResourceType across the full (untagged + tagged) tag list,
+                // instead of one UPDATE per tag. Cuts the in-transaction row-lock acquire chain in half
+                // and avoids cross-tag waits exceeding innodb_lock_wait_timeout under concurrent restores.


can we move this to javadoc (preferably on a new method)

DaanHoogland · 2026-09-01T11:38:20Z

+        // Batched: one UPDATE across all (untagged + tagged) rows for primary_storage. Same rationale
+        // as incrementVolumeResourceCount — reduces in-transaction lock-acquire chain on resize paths.


can we move this to javadoc (preferably on a new method)

kiranchavala

LGTM tested manuallywith a custom shell script

It gives you these functions:

setup: tags the pools, creates the ssd/hdd offerings and the ROOT/d1/d2 domains, and creates account a841 (in d2) and account a841b (in d1).
snap : saves the volume and primary_storage totals, untagged and tagged, for both accounts and for d2, d1 and ROOT.
diffsnap : shows what changed between two snapshots.

dbsnap: shows the raw resource_count rows through mysql. Edit the MYSQL line first.

driftcheck: runs updateResourceCount on ROOT and reports whether any count changed.

deploy_vm [args]: deploys a VM as a841 on the ssd compute offering.

#!/bin/bash # PR 841 test helpers: resource_count batching across storage tags. # Usage: fill in the variables below, then `source pr841-test.sh` and run the steps by hand. # Requires: cmk (synced, admin profile), jq, and a mysql client for the DB checks. CMK="cmk -o json" MYSQL="mysql -u cloud -p<password> -h <db-host> cloud" # edit ZONE="18132ba3-e18a-4fc7-b7f3-8f647000eea2" TPL="32f053ec-b669-11f1-af20-1e00b3000337" NET="b74ccc11-d82c-45ec-bfe8-b12d6e487881" # a shared network the test accounts can use POOL_SSD="617e3db4-5ac5-31c7-9625-e9058dbad1c2" # primary storage to tag 'ssd' POOL_HDD="41743c9a-093d-303f-bc29-33d4b5a1bfaf" # primary storage to tag 'hdd' (optional) ROOT_DOMAIN=$($CMK list domains name=ROOT | jq -r '.domain[0].id') USER_ROLE=$($CMK list roles name=User | jq -r '.role[0].id') # ---------- 1. Setup ---------- setup() { $CMK update configuration name=resource.limit.storage.tags value=ssd,hdd $CMK update storagepool id=$POOL_SSD tags=ssd [ -n "$POOL_HDD" ] && $CMK update storagepool id=$POOL_HDD tags=hdd SO_SSD=$($CMK create serviceoffering name=so-ssd displaytext=so-ssd cpunumber=1 cpuspeed=500 memory=512 tags=ssd | jq -r '.serviceoffering.id') DO_SSD=$($CMK create diskoffering name=do-ssd displaytext=do-ssd disksize=1 tags=ssd | jq -r '.diskoffering.id') DO_SSD_CUSTOM=$($CMK create diskoffering name=do-ssd-custom displaytext=do-ssd-custom customized=true tags=ssd | jq -r '.diskoffering.id') DO_HDD_CUSTOM=$($CMK create diskoffering name=do-hdd-custom displaytext=do-hdd-custom customized=true tags=hdd | jq -r '.diskoffering.id') D1=$($CMK create domain name=d1 | jq -r '.domain.id') D2=$($CMK create domain name=d2 parentdomainid=$D1 | jq -r '.domain.id') $CMK create account username=u841 account=a841 password='Passw0rd!' email=a841@example.com \ firstname=pr lastname=841 roleid=$USER_ROLE domainid=$D2 > /dev/null $CMK create account username=u841b account=a841b password='Passw0rd!' email=a841b@example.com \ firstname=pr lastname=841b roleid=$USER_ROLE domainid=$D1 > /dev/null # target for assignVirtualMachine export SO_SSD DO_SSD DO_SSD_CUSTOM DO_HDD_CUSTOM D1 D2 echo "SO_SSD=$SO_SSD DO_SSD=$DO_SSD DO_SSD_CUSTOM=$DO_SSD_CUSTOM DO_HDD_CUSTOM=$DO_HDD_CUSTOM D1=$D1 D2=$D2" } # ---------- Snapshot of volume / primary_storage counts (account + every parent domain) ---------- _counts='{name, volumetotal, primarystoragetotal, tagged: [.taggedresources[]? | select(.resourcetypename=="volume" or .resourcetypename=="primary_storage") | "\(.resourcetypename)/\(.tag)=\(.total)"]}' snap() { # snap <label> -> writes snap-<label>.json and prints it { $CMK list accounts name=a841 domainid=$D2 details=all | jq -c ".account[0] | $_counts" $CMK list accounts name=a841b domainid=$D1 details=all | jq -c ".account[0] | $_counts" for d in $D2 $D1 $ROOT_DOMAIN; do $CMK list domains id=$d details=all | jq -c ".domain[0] | $_counts" done } | tee "snap-$1.json" } diffsnap() { diff "snap-$1.json" "snap-$2.json"; } # diffsnap before after dbsnap() { # raw rows straight from the DB (volumetotal etc. above come from the same table) $MYSQL -e "SELECT id, account_id, domain_id, type, tag, count FROM resource_count WHERE type IN ('volume','primary_storage') AND (account_id IN (SELECT id FROM account WHERE account_name IN ('a841','a841b')) OR domain_id IN (SELECT id FROM domain WHERE name IN ('d1','d2','ROOT'))) ORDER BY type, tag, account_id, domain_id;" } # Recalculate from the real state; if this changes any count vs the snapshot just before it, the incremental path drifted. driftcheck() { snap pre-recalc > /dev/null $CMK update resourcecount domainid=$ROOT_DOMAIN > /dev/null snap post-recalc > /dev/null if diffsnap pre-recalc post-recalc; then echo "OK: no drift"; else echo "DRIFT: counts changed after updateResourceCount"; fi } deploy_vm() { # deploy_vm <name> [extra args] $CMK deploy virtualmachine name=$1 zoneid=$ZONE templateid=$TPL serviceofferingid=$SO_SSD \ networkids=$NET account=a841 domainid=$D2 "${@:2}" | jq -r '.virtualmachine.id' }

soruce test.sh

step 1: execute setup
setup

Step 2: Check each operation

After each command, run diffsnap. For an ssd resource, the untagged total and the ssd value should change by the same amount on a841, d2, d1 and ROOT. The hdd values must not change. snap s0 VM1=$(deploy_vm vm841a); snap s1; diffsnap s0 s1 # volume +1, primary_storage +root size VM2=$(deploy_vm vm841b diskofferingid=$DO_SSD); snap s2; diffsnap s1 s2 # volume +2, primary_storage +root + 1 GiB save_ids # create + attach a data volume V=$(cmk -o json create volume name=v841 zoneid=$ZONE diskofferingid=$DO_SSD_CUSTOM size=1 account=a841 domainid=$D2 | jq -r '.volume.id') save_ids; snap s3; diffsnap s2 s3 # volume +1, primary_storage +1 GiB cmk attach volume id=$V virtualmachineid=$VM1; snap s4; diffsnap s3 s4 # no change # resize up, then down cmk resize volume id=$V size=3; snap s5; diffsnap s4 s5 # primary_storage +2 GiB cmk resize volume id=$V size=2 shrinkok=true; snap s6; diffsnap s5 s6 # primary_storage -1 GiB # move ssd -> hdd (skip if you have no hdd pool) cmk change offeringforvolume id=$V diskofferingid=$DO_HDD_CUSTOM size=2 automigrate=true snap s7; diffsnap s6 s7 # ssd -1 vol/-2 GiB, hdd +1 vol/+2 GiB, untagged unchanged # hide / show cmk update volume id=$V displayvolume=false; snap s8; diffsnap s7 s8 # counts removed cmk update volume id=$V displayvolume=true; snap s9; diffsnap s8 s9 # counts back # destroy / recover / expunge cmk detach volume id=$V cmk destroy volume id=$V; snap s10; diffsnap s9 s10 # counts removed cmk recover volume id=$V; snap s11; diffsnap s10 s11 # counts back cmk destroy volume id=$V expunge=true; snap s12; diffsnap s11 s12 # counts removed # restore variants (the path the PR targets) cmk restore virtualmachine virtualmachineid=$VM1; snap s13; diffsnap s12 s13 # net 0 cmk restore virtualmachine virtualmachineid=$VM1 rootdisksize=20; snap s14; diffsnap s13 s14 # primary_storage +(20 GiB - old root) # optional, only if you have a second template (TPL2): cmk restore virtualmachine virtualmachineid=$VM1 templateid=$TPL2; snap s15; diffsnap s14 s15 # primary_storage +(new root - old root) driftcheck # must print "OK: no drift"

Step 3: One UPDATE per type (MySQL general log)

Run this on the DB host as MySQL root:

mysql -u root -p -e "SET GLOBAL log_output='FILE'; SET GLOBAL general_log_file='/var/lib/mysql/cs-gl.log'; SET GLOBAL general_log=ON;" cmk restore virtualmachine virtualmachineid=$VM1 mysql -u root -p -e "SET GLOBAL general_log=OFF;"

grep -i 'UPDATE cloud.resource_count' /var/lib/mysql/cs-gl.log

[root@ref-trl-12391-k-Mol8-kiran-chavala-mgmt1 ~]# ls -l /tmp/systemd-private-*-mysqld.service-*/tmp/cs-gl.log -rw-r-----. 1 mysql mysql 7210025 Sep 24 13:35 /tmp/systemd-private-922a428aaa014dc699a385a62169a65d-mysqld.service-2euny4/tmp/cs-gl.log [root@ref-trl-12391-k-Mol8-kiran-chavala-mgmt1 ~]# grep -i 'UPDATE `cloud`.`resource_count`' /tmp/systemd-private-922a428aaa014dc699a385a62169a65d-mysqld.service-2euny4/tmp/cs-gl.log 2026-09-24T13:30:06.436043Z 6407 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (129,99,67,83,100,68,84,11) 2026-09-24T13:30:06.436074Z 6407 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 1 WHERE `id` IN (129,99,67,83,100,68,84,11) 2026-09-24T13:30:06.441130Z 6407 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (130,3,109,77,93,110,78,94) 2026-09-24T13:30:06.441179Z 6407 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 8589934592 WHERE `id` IN (130,3,109,77,93,110,78,94) 2026-09-24T13:30:06.458283Z 6407 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (97,65,81,9) 2026-09-24T13:30:06.458319Z 6407 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 1 WHERE `id` IN (97,65,81,9) 2026-09-24T13:30:06.460821Z 6407 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (1,107,75,91) 2026-09-24T13:30:06.460867Z 6407 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 1 WHERE `id` IN (1,107,75,91) 2026-09-24T13:30:06.463357Z 6407 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (2,108,76,92) 2026-09-24T13:30:06.463389Z 6407 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 512 WHERE `id` IN (2,108,76,92) 2026-09-24T13:32:31.632347Z 6408 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (129,99,67,83,100,68,84,11) 2026-09-24T13:32:31.632399Z 6408 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 1 WHERE `id` IN (129,99,67,83,100,68,84,11) 2026-09-24T13:32:31.639208Z 6408 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` + ? WHERE `id` IN (130,3,109,77,93,110,78,94) 2026-09-24T13:32:31.639273Z 6408 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` + 8589934592 WHERE `id` IN (130,3,109,77,93,110,78,94) 2026-09-24T13:32:31.679208Z 6408 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` - ? WHERE `id` IN (129,99,67,83,100,68,84,11) 2026-09-24T13:32:31.679245Z 6408 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` - 1 WHERE `id` IN (129,99,67,83,100,68,84,11) 2026-09-24T13:32:31.685030Z 6408 Prepare UPDATE `cloud`.`resource_count` SET `count` = `count` - ? WHERE `id` IN (130,3,109,77,93,110,78,94) 2026-09-24T13:32:31.685070Z 6408 Execute UPDATE `cloud`.`resource_count` SET `count` = `count` - 8589934592 WHERE `id` IN (130,3,109,77,93,110,78,94)

What you should see: each UPDATE for volume and primary_storage (increment and decrement) lists several IDs in IN (…): a841, d2, d1 and ROOT, for both untagged and ssd. The pre-PR build sends one UPDATE per tag, so twice as many.

Step 4: Concurrency with two VMs

Repeated parallel restores. The first two restores reset the root sizes left over from step 2, so they don't show up in the diff.

$MYSQL -e "SET GLOBAL innodb_lock_wait_timeout=5;" cmk restore virtualmachine virtualmachineid=$VM1 > /dev/null; cmk restore virtualmachine virtualmachineid=$VM2 > /dev/null snap loop-before for vm in $VM1 $VM2; do ( for i in $(seq 1 2); do cmk restore virtualmachine virtualmachineid=$vm > /dev/null 2>&1 || echo "FAIL $vm #$i"; done ) & done; wait snap loop-after; diffsnap loop-before loop-after # expect no change grep -c "Lock wait timeout exceeded" /var/log/cloudstack/management/management-server.log 0 driftcheck

Damans227 · 2026-10-15T14:26:23Z

+                reservationDao.removeByIds(reservationIds);
+                Set<Long> rowIds = collectRowIdsForTags(accountId, type, tags, numToIncrement, true);
+                if (rowIds.isEmpty()) {
+                    logger.warn("No resource_count rows resolved to increment for Account = " + accountId


any reason not to throw here like before? when no count rows are found the create used to fail, now it only logs and carries on

ganeshmaharaj and others added 3 commits August 18, 2026 20:54

Address review comments

0eaa46a

Fix build

eb97e4b

nvazquez added this to the 4.22.2 milestone Aug 19, 2026

nvazquez requested review from shwstppr, sureshanaparti and vishesh92 August 19, 2026 00:57

nvazquez added the enhancement label Aug 19, 2026

DaanHoogland added this to CloudStack Testing Aug 31, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 31, 2026

DaanHoogland reviewed Sep 1, 2026

View reviewed changes

DaanHoogland moved this from Backlog to Ready in CloudStack Testing Sep 1, 2026

DaanHoogland removed the enhancement label Sep 5, 2026

kiranchavala approved these changes Sep 25, 2026

View reviewed changes

Damans227 reviewed Oct 15, 2026

View reviewed changes

Navigation Menu

Repository navigation

Batch resource_count UPDATEs across tags to reduce lock contention - #13915

Batch resource_count UPDATEs across tags to reduce lock contention#13915
nvazquez wants to merge 3 commits into
apache:4.22from
shapeblue:422-batch-resourcecount-update-across-tags

nvazquez commented Aug 19, 2026

Uh oh!

nvazquez commented Aug 19, 2026

Uh oh!

blueorangutan commented Aug 19, 2026

Uh oh!

codecov Bot commented Aug 19, 2026 •

edited

Loading

Uh oh!

blueorangutan commented Aug 19, 2026

Uh oh!

DaanHoogland Sep 1, 2026

Uh oh!

DaanHoogland Sep 1, 2026

Uh oh!

kiranchavala left a comment

Uh oh!

Damans227 Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		// Batched: one UPDATE across all (untagged + tagged) rows for primary_storage. Same rationale
		// as incrementVolumeResourceCount — reduces in-transaction lock-acquire chain on resize paths.

Conversation

nvazquez commented Aug 19, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

nvazquez commented Aug 19, 2026

Uh oh!

blueorangutan commented Aug 19, 2026

Uh oh!

codecov Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Aug 19, 2026

Uh oh!

DaanHoogland Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

kiranchavala left a comment

Choose a reason for hiding this comment

Uh oh!

Damans227 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov Bot commented Aug 19, 2026 •

edited

Loading