Repository navigation
Conversation
* Store last successful MS host for agent reconnection fallback * Add newline at end of CapacityManagerImplTest.java
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.20 #14002 +/- ##
============================================
+ Coverage 16.32% 16.33% +0.01%
- Complexity 13556 13564 +8
============================================
Files 5669 5669
Lines 501399 501425 +26
Branches 60902 60904 +2
============================================
+ Hits 81847 81913 +66
+ Misses 410390 410341 -49
- Partials 9162 9171 +9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
it seems we have been working in the same area @Pearl1594 , #14000 is about agent rebalancing for local agents and #14001 about rebalancing remote agents. Can you have a look at those as well, please? |
|
It seems so - but they don't clash. I can have a look at them. Thanks @DaanHoogland |
|
|
@blueorangutan package |
|
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
DaanHoogland
moved this from Backlog
to conflict/waiting for author
in CloudStack Testing
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19030 |
There was a problem hiding this comment.
LGTM Tested manually with 2 ms
The parameter last.setup.completed.host is present in agent.properties
grep -n "^last.setup.completed.host" /etc/cloudstack/agent/agent.properties
Test A — the first-connect
1. systemctl stop cloudstack-agent
2. sed -i '/^last.setup.completed.host/d' /etc/cloudstack/agent/agent.properties
3.systemctl start cloudstack-agent
4. sleep 45
5. grep -c "^last.setup.completed.host" /etc/cloudstack/agent/agent.properties
Confirm the agent actually reached Ready before judging:
6. grep -n "Ready command is processed for agent" /var/log/cloudstack/agent/agent.log | tail -2
[root@ref-trl-12367-k-Mol8-kiran-chavala-kvm2 ~]# grep -n "Ready command is processed for agent" /var/log/cloudstack/agent/agent.log | tail -2
22778:2026-09-16 12:13:38,188 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:) Ready command is processed for agent [id: 2, uuid: 568a5362-6aa9-4b64-a9e7-786086ba2c93, name: ref-trl-12367-k-Mol8-kiran-chavala-kvm2]
22804:2026-09-16 12:13:39,613 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Ready command is processed for agent [id: 2, uuid: 568a5362-6aa9-4b64-a9e7-786086ba2c93, name: ref-trl-12367-k-Mol8-kiran-chavala-kvm2]
Test B — Break it centrally:
-
cmk list configurations name=host filter=name,value
-
cmk update configuration name=host value=10.99.99.99
-
Confirm propagation reached the agent:
MS side
[root@ref-trl-12367-k-Mol8-kiran-chavala-mgmt1 ~]# grep -n "Propagating management server list update to agents" /var/log/cloudstack/management/management-server.log
27017:2026-09-16 08:29:16,649 DEBUG [o.a.c.a.l.IndirectAgentLBServiceImpl] (qtp253011924-18:[ctx-72433d19, ctx-79e03618, ctx-882c796e]) (logid:86a3a74f) Propagating management server list update to agents
agent side
[root@ref-trl-12367-k-Mol8-kiran-chavala-kvm2 ~]#grep -n "Processed new management server list" /var/log/cloudstack/agent/agent.log
27000:2026-09-16 12:48:16,739 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:) Processed new management server list: 10.99.99.99@static
[root@ref-trl-12367-k-Mol8-kiran-chavala-kvm1 ~]# grep "^host=\|^last.setup.completed.host" /etc/cloudstack/agent/agent.properties
last.setup.completed.host=10.0.35.19
host=10.99.99.99@static
systemctl restart cloudstack-agent
tail -f /var/log/cloudstack/agent/agent.log
Expected sequence with the patch:
Connecting to host: 10.99.99.99 (Agent.java:332)
IOException while connecting to 10.99.99.99:8250 (NioClient.java:92)
Connecting to host: ← the appended fallback
Connected to the host: (Agent.java:662)
Ready command is processed for agent ...
The host returns to Up with no SSH intervention — that's the pass condition. Witn 10.99.99.99 indefinitely.
2026-09-16 12:51:11,843 DEBUG [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Multipath Cleanup Job elapsed time (ms): 30; result: 0
2026-09-16 12:51:23,321 ERROR [utils.nio.NioClient] (main:[]) (logid:) IOException while connecting to 10.99.99.99:8250 java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:591)
at java.base/sun.nio.ch.Net.connect(Net.java:598)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853)
at com.cloud.utils.nio.NioClient.init(NioClient.java:65)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112)
at com.cloud.agent.Agent.start(Agent.java:322)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
at com.cloud.agent.AgentShell.start(AgentShell.java:559)
at com.cloud.agent.AgentShell.main(AgentShell.java:589)
2026-09-16 12:51:23,323 WARN [utils.nio.NioClient] (main:[]) (logid:) Unable to connect to remote: is there a server running on port 8250? java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:591)
at java.base/sun.nio.ch.Net.connect(Net.java:598)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853)
at com.cloud.utils.nio.NioClient.init(NioClient.java:65)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112)
at com.cloud.agent.Agent.start(Agent.java:322)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
at com.cloud.agent.AgentShell.start(AgentShell.java:559)
at com.cloud.agent.AgentShell.main(AgentShell.java:589)
2026-09-16 12:51:23,324 WARN [cloud.agent.Agent] (main:[]) (logid:) Attempt to connect to server generated NIO Connection Exception Connection timed out, trying again
2026-09-16 12:51:28,326 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [ssl.handshake.timeout] has empty or null value. Using default value [null].
2026-09-16 12:51:28,326 INFO [cloud.agent.Agent] (main:[]) (logid:) Connecting to host: 10.0.35.171
2026-09-16 12:51:28,326 INFO [utils.nio.NioClient] (main:[]) (logid:) Connecting to 10.0.35.171:8250
2026-09-16 12:51:28,327 INFO [utils.nio.NioClient] (main:[]) (logid:) Connected to 10.0.35.171:8250
2026-09-16 12:51:28,330 INFO [utils.nio.Link] (main:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2026-09-16 12:51:28,872 INFO [utils.nio.NioClient] (main:[]) (logid:) SSL: Handshake done
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19274 |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian Build Failed (tid-17020) |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17080)
|
| final boolean alreadyPresent = Arrays.stream(_host.split(",")) | ||
| .map(String::trim) | ||
| .anyMatch(candidate::equalsIgnoreCase); | ||
| host = alreadyPresent ? _host : _host + "," + candidate; |
There was a problem hiding this comment.
should we log a warning when the agent ends up on this fallback host? otherwise the hosts show up again and nobody notices the host list is still wrong
This branch has not been deployed
7 participants
Description
The issue is that once a bad hosts configuration update made, there's no way to get the Agents back in Up state without SSH-ing on each of them and manually updating the agent.properties file (because they're not listening to any new update from a control plane, since they're not connected anymore).
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?