Skip to content

[DNM] CS10 watcher edpm s2i jobs - #450

Draft
raukadah wants to merge 1 commit into
openstack-k8s-operators:mainfrom
raukadah:cs10
Draft

raukadah wants to merge 1 commit into
openstack-k8s-operators:mainfrom
raukadah:cs10

Conversation

@raukadah

Copy link
Copy Markdown
Contributor

@openshift-ci

openshift-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci

openshift-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign seanmooney for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Zuul encountered a syntax error while parsing its
configuration in the repo openstack-k8s-operators/watcher-operator on branch main. The
problem was:

The nodeset "centos-10-medium-2x-centos-10-crc-cloud-ocp-4-20-1-3xl"
was not found.

The problem appears in the "watcher-operator-base-centos-10" job stanza:

job:
name: watcher-operator-base-centos-10
nodeset: centos-10-medium-2x-centos-10-crc-cloud-ocp-4-20-1-3xl
parent: podified-multinode-edpm-deployment-crc-2comp-centos-10
description: |
A multinode EDPM Zuul job which has one ansible controller, one
extracted crc and two computes. It will be used for testing watcher-operator.
Configures Nova and Cinder to send notifications over a dedicated rabbitmq instance.
vars:
...

in "openstack-k8s-operators/watcher-operator/.zuul.yaml@main", line 22

@centosinfra-prod-github-app

Copy link
Copy Markdown

Zuul encountered a syntax error while parsing its
configuration in the repo openstack-k8s-operators/watcher-operator on branch main. The
problem was:

The nodeset "centos-10-medium-2x-centos-10-crc-cloud-ocp-4-20-1-3xl"
was not found.

The problem appears in the "watcher-operator-base-centos-10" job stanza:

job:
name: watcher-operator-base-centos-10
nodeset: centos-10-medium-2x-centos-10-crc-cloud-ocp-4-20-1-3xl
parent: podified-multinode-edpm-deployment-crc-2comp-centos-10
description: |
A multinode EDPM Zuul job which has one ansible controller, one
extracted crc and two computes. It will be used for testing watcher-operator.
Configures Nova and Cinder to send notifications over a dedicated rabbitmq instance.
vars:
...

in "openstack-k8s-operators/watcher-operator/.zuul.yaml@main", line 22

@centosinfra-prod-github-app

Copy link
Copy Markdown

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#4163 is needed.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/bc6989466eca411a952ffdba814114e0

✔️ openstack-k8s-operators-content-provider SUCCESS in 45m 15s
watcher-operator-validation-master NODE_FAILURE Node(set) request 099-0000199960 failed in 0s
✔️ noop SUCCESS in 0s
watcher-operator-kuttl NODE_FAILURE Node(set) request 099-0000199961 failed in 0s
✔️ watcher-s2i-content-provider SUCCESS in 45m 13s (non-voting)
watcher-s2i-tempest NODE_FAILURE Node(set) request 099-0000199962 failed in 0s (non-voting)

@raukadah

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/158e05b94974444ebf9b69ffbfbb2133

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 22m 28s
✔️ watcher-operator-validation-master SUCCESS in 2h 06m 39s
✔️ noop SUCCESS in 0s
✔️ watcher-operator-kuttl SUCCESS in 54m 05s
✔️ watcher-s2i-content-provider SUCCESS in 2h 01m 54s (non-voting)
watcher-s2i-tempest POST_FAILURE in 1h 45m 40s (non-voting)

@raukadah

Copy link
Copy Markdown
Contributor Author

most of the live migration tests are failing https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs//e8a/rdoproject.org/e8ae786d8cef40af8e26e93a8dcebece/controller/ci-framework-data/tests/test_operator/tempest-tests-tempest/tempest_results.html.gz

One of the example of zone migration test

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/watcher_tempest_plugin/tests/scenario/test_execute_zone_migration.py", line 82, in test_execute_zone_migration_with_destination_host
    self.execute_action_plan_and_validate_states(action_plan['uuid'])
  File "/usr/lib/python3.12/site-packages/watcher_tempest_plugin/tests/scenario/base.py", line 1090, in execute_action_plan_and_validate_states
    self.assertIn(finished_ap['state'], expected_action_plan_states)
  File "/usr/lib/python3.12/site-packages/testtools/testcase.py", line 522, in assertIn
    self.assertThat(haystack, Contains(needle), message)
  File "/usr/lib/python3.12/site-packages/testtools/testcase.py", line 704, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'FAILED' not in ['SUCCEEDED']

Based on compute-0 log in sosreport-compute-0/sosreport-compute-0-2026-09-15-vliyjop/var/log/messages

  Sep 15 05:17:38 np0000179866 nova_compute[169998]: 2026-09-15 05:17:38.952 170002 ERROR nova.virt.libvirt.driver [None req-1214097c-6e5f-4815-bb20-6d2e27a7e4b2       
 59617e1814ab48a3a76ee604cb5c2d1b ad090f5bec4e4a28a5a3f34861fb0756 - - default default] [instance: e6dfa97f-bcaf-4a18-97e0-f857f67b02ce] Live Migration failure: unable  
 to connect to server at 'compute-1.ctlplane.example.com:16514': Connection refused: libvirt.libvirtError: unable to connect to server at                                
 'compute-1.ctlplane.example.com:16514': Connection refused   

It might be network connectvity issue between two nodes.

@raukadah

Copy link
Copy Markdown
Contributor Author

recheck after node hold

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/67a931e293e54d3681a401c6df13f15f

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 03s
watcher-operator-validation-master FAILURE in 1h 55m 05s
✔️ noop SUCCESS in 0s
✔️ watcher-operator-kuttl SUCCESS in 51m 54s
✔️ watcher-s2i-content-provider SUCCESS in 2h 01m 09s (non-voting)
watcher-s2i-tempest POST_FAILURE in 1h 46m 25s (non-voting)

@raukadah

Copy link
Copy Markdown
Contributor Author

Sep 15 05:17:38 np0000179866 nova_compute[169998]: 2026-09-15 05:17:38.952 170002 ERROR nova.virt.libvirt.driver [None req-1214097c-6e5f-4815-bb20-6d2e27a7e4b2
59617e1814ab48a3a76ee604cb5c2d1b ad090f5bec4e4a28a5a3f34861fb0756 - - default default] [instance: e6dfa97f-bcaf-4a18-97e0-f857f67b02ce] Live Migration failure: unable
to connect to server at 'compute-1.ctlplane.example.com:16514': Connection refused: libvirt.libvirtError: unable to connect to server at
'compute-1.ctlplane.example.com:16514': Connection refused

More logs:

[zuul@compute-1 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                    COMMAND               CREATED      STATUS                PORTS                                     NAMES
2770ed279325  quay.io/podified-antelope-centos9/openstack-ovn-controller:current-podified              /usr/bin/ovn-cont...  2 hours ago  Up 2 hours (healthy)                                            ovn_controller
e6edca5c8448  quay.io/podified-antelope-centos9/openstack-neutron-metadata-agent-ovn:current-podified  /usr/bin/neutron-...  2 hours ago  Up 2 hours (healthy)                                            ovn_metadata_agent
a6ca6042eec2  quay.io/podified-antelope-centos9/openstack-nova-compute:current-podified                /usr/bin/nova-com...  2 hours ago  Up 2 hours                                                      nova_compute
87d2a09e2218  quay.io/openstack-k8s-operators/prometheus-podman-exporter:latest                        --web.config.file...  2 hours ago  Up 2 hours (healthy)  192.168.122.101:9882->9882/tcp            podman_exporter
208800b5d473  quay.io/openstack-k8s-operators/openstack-network-exporter:current-podified              /app/openstack-ne...  2 hours ago  Up 2 hours (healthy)  192.168.122.101:9105->9105/tcp, 1981/tcp  openstack_network_exporter
[zuul@compute-1 ~]$ rpm -qa | grep -E "libvirt-daemon|qemu-kvm" | sort
libvirt-daemon-12.5.0-4.el10.x86_64
libvirt-daemon-common-12.5.0-4.el10.x86_64
libvirt-daemon-config-network-12.5.0-4.el10.x86_64
libvirt-daemon-config-nwfilter-12.5.0-4.el10.x86_64
libvirt-daemon-driver-interface-12.5.0-4.el10.x86_64
libvirt-daemon-driver-network-12.5.0-4.el10.x86_64
libvirt-daemon-driver-nodedev-12.5.0-4.el10.x86_64
libvirt-daemon-driver-nwfilter-12.5.0-4.el10.x86_64
libvirt-daemon-driver-qemu-12.5.0-4.el10.x86_64
libvirt-daemon-driver-secret-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-core-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-disk-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-iscsi-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-logical-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-mpath-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-rbd-12.5.0-4.el10.x86_64
libvirt-daemon-driver-storage-scsi-12.5.0-4.el10.x86_64
libvirt-daemon-lock-12.5.0-4.el10.x86_64
libvirt-daemon-log-12.5.0-4.el10.x86_64
libvirt-daemon-plugin-lockd-12.5.0-4.el10.x86_64
libvirt-daemon-proxy-12.5.0-4.el10.x86_64
qemu-kvm-10.1.0-28.el10.x86_64
qemu-kvm-audio-pa-10.1.0-28.el10.x86_64
qemu-kvm-block-blkio-10.1.0-28.el10.x86_64
qemu-kvm-block-rbd-10.1.0-28.el10.x86_64
qemu-kvm-common-10.1.0-28.el10.x86_64
qemu-kvm-core-10.1.0-28.el10.x86_64
qemu-kvm-device-display-virtio-gpu-10.1.0-28.el10.x86_64
qemu-kvm-device-display-virtio-gpu-pci-10.1.0-28.el10.x86_64
qemu-kvm-device-display-virtio-vga-10.1.0-28.el10.x86_64
qemu-kvm-device-usb-host-10.1.0-28.el10.x86_64
qemu-kvm-device-usb-redirect-10.1.0-28.el10.x86_64
qemu-kvm-docs-10.1.0-28.el10.x86_64
qemu-kvm-tools-10.1.0-28.el10.x86_64
qemu-kvm-ui-egl-headless-10.1.0-28.el10.x86_64
qemu-kvm-ui-opengl-10.1.0-28.el10.x86_64
[zuul@compute-1 ~]$ sudo systemctl status virtqemud virtproxyd virtproxyd-tls.socket libvirtd libvirtd-tls.socket
● virtqemud.service - libvirt QEMU daemon
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-09-16 06:45:44 UTC; 2h 17min ago
 Invocation: 754e10e12889452b835665723d7f1a31
TriggeredBy: ● virtqemud.socket
             ● virtqemud-ro.socket
             ● virtqemud-admin.socket
       Docs: man:virtqemud(8)
             https://libvirt.org/
   Main PID: 168795 (virtqemud)
         IO: 20K read, 503K written
      Tasks: 20 (limit: 32768)
     Memory: 21.5M (peak: 44M)
        CPU: 4.351s
     CGroup: /system.slice/virtqemud.service
             └─168795 /usr/sbin/virtqemud --timeout 120

Sep 16 08:54:04 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 08:55:05 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 08:56:06 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 08:57:07 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 08:58:09 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 08:59:09 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 09:00:09 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 09:01:09 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 09:02:10 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory
Sep 16 09:03:11 compute-1 virtqemud[168795]: Failed to connect socket to '/var/run/libvirt/virtnodedevd-sock': No such file or directory

○ virtproxyd.service - libvirt proxy daemon
     Loaded: loaded (/usr/lib/systemd/system/virtproxyd.service; enabled; preset: disabled)
     Active: inactive (dead) since Wed 2026-09-16 08:56:00 UTC; 7min ago
   Duration: 2min 19ms
 Invocation: bcf54a74800649efbe94589d5fbb5ffe
TriggeredBy: ● virtproxyd-admin.socket
             ● virtproxyd-ro.socket
             ● virtproxyd.socket
             ● virtproxyd-tls.socket
       Docs: man:virtproxyd(8)
             https://libvirt.org/
    Process: 222588 ExecStart=/usr/sbin/virtproxyd $VIRTPROXYD_ARGS (code=exited, status=0/SUCCESS)
   Main PID: 222588 (code=exited, status=0/SUCCESS)
         IO: 0B read, 0B written
   Mem peak: 5M
        CPU: 51ms

Sep 16 08:54:00 compute-1 systemd[1]: Starting virtproxyd.service - libvirt proxy daemon...
Sep 16 08:54:00 compute-1 systemd[1]: Started virtproxyd.service - libvirt proxy daemon.
Sep 16 08:54:00 compute-1 virtproxyd[222588]: libvirt version: 12.5.0, package: 4.el10 (builder@centos.org, 2026-08-27-13:49:42, )
Sep 16 08:54:00 compute-1 virtproxyd[222588]: hostname: compute-1, uid: 0
Sep 16 08:54:00 compute-1 virtproxyd[222588]: authentication failed: TLS handshake failed The TLS connection was non-properly terminated.
Sep 16 08:54:13 compute-1 virtproxyd[222588]: authentication failed: TLS handshake failed The TLS connection was non-properly terminated.
Sep 16 08:56:00 compute-1 systemd[1]: virtproxyd.service: Deactivated successfully.

● virtproxyd-tls.socket - libvirt proxy daemon TLS IP socket
     Loaded: loaded (/usr/lib/systemd/system/virtproxyd-tls.socket; enabled; preset: disabled)
     Active: active (listening) since Wed 2026-09-16 08:47:24 UTC; 15min ago
 Invocation: a398118800d74059ac1d72728b9a5604
   Triggers: ● virtproxyd.service
     Listen: [::]:16514 (Stream)
         IO: 0B read, 0B written
      Tasks: 0 (limit: 48527)
     Memory: 8K (peak: 256K)
        CPU: 766us
     CGroup: /system.slice/virtproxyd-tls.socket

Sep 16 08:47:24 compute-1 systemd[1]: Listening on virtproxyd-tls.socket - libvirt proxy daemon TLS IP socket.

○ libvirtd.service
     Loaded: masked (Reason: Unit libvirtd.service is masked.)
    Drop-In: /usr/lib/systemd/system/libvirtd.service.d
             └─10-secret.conf
     Active: inactive (dead)
TriggeredBy: ○ libvirtd-ro.socket
             ○ libvirtd-admin.socket
             ○ libvirtd.socket

○ libvirtd-tls.socket
     Loaded: masked (Reason: Unit libvirtd-tls.socket is masked.)
     Active: inactive (dead)
[zuul@compute-1 ~]$ sudo ss -tulpn | grep 16514
tcp   LISTEN 0      1000               *:16514            *:*    users:(("systemd",pid=1,fd=101))

@raukadah

Copy link
Copy Markdown
Contributor Author

quay.io/podified-antelope-centos9/openstack-nova-compute:current-podified

[zuul@compute-1 ~]$ sudo podman exec nova_compute rpm -qa | grep -i libvirt
libvirt-libs-11.10.0-14.el9.x86_64
libvirt-client-11.10.0-14.el9.x86_64
libvirt-daemon-common-11.10.0-14.el9.x86_64
libvirt-daemon-driver-secret-11.10.0-14.el9.x86_64
libvirt-daemon-driver-storage-core-11.10.0-14.el9.x86_64
libvirt-daemon-driver-nwfilter-11.10.0-14.el9.x86_64
libvirt-daemon-log-11.10.0-14.el9.x86_64
python3-libvirt-11.10.0-2.el9.x86_64
libvirt-daemon-driver-nodedev-11.10.0-14.el9.x86_64
libvirt-daemon-driver-qemu-11.10.0-14.el9.x86_64
[zuul@compute-1 ~]$ sudo podman exec nova_compute virsh --version
11.10.0

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/dd9c025151a84a59995d6a1ddc1c570b

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 31m 11s
watcher-operator-validation-master FAILURE in 1h 58m 46s
✔️ noop SUCCESS in 0s
✔️ watcher-operator-kuttl SUCCESS in 56m 27s
✔️ watcher-s2i-content-provider SUCCESS in 2h 44m 57s (non-voting)
watcher-s2i-tempest POST_FAILURE in 1h 26m 55s (non-voting)

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d9dd484cd0d9438a9755335fc810a242

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 20m 45s
✔️ watcher-operator-validation-master SUCCESS in 2h 04m 05s
✔️ noop SUCCESS in 0s
✔️ watcher-operator-kuttl SUCCESS in 56m 47s
✔️ watcher-s2i-content-provider SUCCESS in 2h 05m 00s (non-voting)
watcher-s2i-tempest FAILURE in 1h 48m 30s (non-voting)

@centosinfra-prod-github-app

Copy link
Copy Markdown

Zuul encountered a syntax error while parsing its
configuration in the repo openstack-k8s-operators/watcher-operator on branch main. The
problem was:

The nodeset "centos-10-medium-2x-centos-10-crc-cloud-ocp-4-20-1-3xl"
was not found.

The problem appears in the "watcher-operator-base-centos-10" job stanza:

job:
name: watcher-operator-base-centos-10
nodeset: centos-10-medium-2x-centos-10-crc-cloud-ocp-4-20-1-3xl
parent: podified-multinode-edpm-deployment-crc-2comp-centos-10
description: |
A multinode EDPM Zuul job which has one ansible controller, one
extracted crc and two computes. It will be used for testing watcher-operator.
Configures Nova and Cinder to send notifications over a dedicated rabbitmq instance.
vars:
...

in "openstack-k8s-operators/watcher-operator/.zuul.yaml@main", line 22

Depends-On: openstack-k8s-operators/ci-framework#4163

Signed-off-by: Chandan <raukadah@gmail.com>
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5e95b050dacc4a0d84ce60a42517035f

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 24m 39s
✔️ watcher-operator-validation-master SUCCESS in 2h 07m 29s
✔️ noop SUCCESS in 0s
✔️ watcher-operator-kuttl SUCCESS in 52m 15s
✔️ watcher-s2i-content-provider SUCCESS in 2h 57m 45s (non-voting)
watcher-s2i-tempest FAILURE in 1h 38m 59s (non-voting)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant