INTEROP-9455: Replace cucushift health check with interop-opp-wait-mcp in OPP configs

[redhat-chai-bot]

Summary

Replace cucushift-installer-check-cluster-health with interop-opp-wait-mcp in the RedHatQE/interop-testing OPP ci-operator configs for OCP 4.22, 5.0, and 5.1.

Problem

The OPP interop tests under RedHatQE/interop-testing consistently fail at the cucushift-installer-check-cluster-health step. When ODF adds worker nodes post-install (e.g. 6→9 nodes), the 99-worker-generated-kubelet MachineConfig is regenerated, triggering a worker MCP rolling update. The generic cucushift health check has a tight 20-check budget with a consecutive-pass requirement that is routinely exhausted before the rollout completes.

Fix

The interop-opp-wait-mcp step (already in the step registry and already used in the stolostron/policy-collection OPP configs since PR #84645) handles this correctly with:

• 90s settle delay before first poll (lets MachineConfig rendering propagate)
• 60-min timeout (vs ~20-min budget in cucushift)
• Parallel rollout (MCP_MAX_UNAVAILABLE=3 — 3 nodes at a time instead of 1)
• 3 consecutive stability checks required

Changes

• RedHatQE-interop-testing-master__opp--ocp-4.22-lpMainline-lp-interop.yaml — removed duplicate cucushift chain (wait-mcp ref already existed)
• RedHatQE-interop-testing-master__opp--ocp-5.0-lpMainline-lp-interop.yaml — replaced 2 occurrences (AWS + vSphere)
• RedHatQE-interop-testing-master__opp--ocp-5.1-lpMainline-lp-interop.yaml — replaced 2 occurrences (AWS + vSphere)

Validated with make jobs — no job file regeneration needed.

---

AI-generated. Review for accuracy.

@amp-rh requested via Chai Bot

Summary by CodeRabbit

This PR updates RedHatQE/interop-testing OPP CI configurations for OCP 4.22, 5.0, and 5.1.

The affected AWS and vSphere jobs replace cucushift-installer-check-cluster-health with interop-opp-wait-mcp. This change updates post-install validation to wait for MachineConfigPool stabilization.

Validation with make jobs is reported as complete.

[openshift-ci-robot]

@redhat-chai-bot: This pull request references INTEROP-9455 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.1.0" version, but no target version was set.

Details

In response to this:

Summary

Replace cucushift-installer-check-cluster-health with interop-opp-wait-mcp in the RedHatQE/interop-testing OPP ci-operator configs for OCP 4.22, 5.0, and 5.1.

Problem

The OPP interop tests under RedHatQE/interop-testing consistently fail at the cucushift-installer-check-cluster-health step. When ODF adds worker nodes post-install (e.g. 6→9 nodes), the 99-worker-generated-kubelet MachineConfig is regenerated, triggering a worker MCP rolling update. The generic cucushift health check has a tight 20-check budget with a consecutive-pass requirement that is routinely exhausted before the rollout completes.

Fix

The interop-opp-wait-mcp step (already in the step registry and already used in the stolostron/policy-collection OPP configs since PR #84645) handles this correctly with:

• 90s settle delay before first poll (lets MachineConfig rendering propagate)
• 60-min timeout (vs ~20-min budget in cucushift)
• Parallel rollout (MCP_MAX_UNAVAILABLE=3 — 3 nodes at a time instead of 1)
• 3 consecutive stability checks required

Changes

• RedHatQE-interop-testing-master__opp--ocp-4.22-lpMainline-lp-interop.yaml — removed duplicate cucushift chain (wait-mcp ref already existed)
• RedHatQE-interop-testing-master__opp--ocp-5.0-lpMainline-lp-interop.yaml — replaced 2 occurrences (AWS + vSphere)
• RedHatQE-interop-testing-master__opp--ocp-5.1-lpMainline-lp-interop.yaml — replaced 2 occurrences (AWS + vSphere)

Validated with make jobs — no job file regeneration needed.

---

AI-generated. Review for accuracy.

@amp-rh requested via Chai Bot

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: 534543b7-9fbf-4e59-9c6b-8e35b0ae2547

📥 Commits

Reviewing files that changed from the base of the PR and between 38aab9f and c81010c.

📒 Files selected for processing (3)

• ci-operator/config/RedHatQE/interop-testing/RedHatQE-interop-testing-master__opp--ocp-4.22-lpMainline-lp-interop.yaml
• ci-operator/config/RedHatQE/interop-testing/RedHatQE-interop-testing-master__opp--ocp-5.0-lpMainline-lp-interop.yaml
• ci-operator/config/RedHatQE/interop-testing/RedHatQE-interop-testing-master__opp--ocp-5.1-lpMainline-lp-interop.yaml

💤 Files with no reviewable changes (1)

• ci-operator/config/RedHatQE/interop-testing/RedHatQE-interop-testing-master__opp--ocp-4.22-lpMainline-lp-interop.yaml

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

Walkthrough

The interop configurations update full-stack test sequences. OCP 4.22 removes the generic cluster-health chain for AWS. OCP 5.0 and 5.1 replace that chain with interop-opp-wait-mcp for AWS and vSphere.

Changes

OPP interop test steps

Layer / File(s)	Summary
OCP 4.22 test-step removal ci-operator/config/RedHatQE/interop-testing/...ocp-4.22...yaml	The AWS full-stack test removes cucushift-installer-check-cluster-health.
OCP 5.0 and 5.1 MCP wait replacement ci-operator/config/RedHatQE/interop-testing/...ocp-5.0...yaml, ...ocp-5.1...yaml	The AWS and vSphere full-stack tests replace cucushift-installer-check-cluster-health with interop-opp-wait-mcp.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: amiskin94

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	The PR activates interop-opp-wait-mcp in the OCP 5.0 and 5.1 AWS and vSphere jobs. Its existing command script uses set -x and writes raw cluster diagnostics to job logs, including `oc get nodes -…	Sanitize the replacement step before enabling it in these jobs. Disable set -x, avoid logging raw oc get nodes -o wide and oc describe mcp output, and report only aggregate health results or explicitly redacted identifiers. Apply the …

✅ Passed checks (14 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: replacing the cucushift health check with interop-opp-wait-mcp in OPP configurations.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PASS: The pull request changes only three static ci-operator YAML step lists. The diff adds or retains the literal ref interop-opp-wait-mcp and removes a chain reference. No It(), Describe(), `C…
Test Structure And Quality	✅ Passed	PASS: The pull request changes only three ci-operator YAML configuration files. The patch contains no Ginkgo test code, It blocks, fixtures, waits, or assertions. Therefore the five Ginkgo test qualit…
Microshift Test Compatibility	✅ Passed	The check is not applicable. The pull request changes only three YAML ci-operator configuration files. It adds no Go files, Ginkgo declarations, or new e2e tests. Therefore, it introduces no MicroShif…
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PASS: The pull request changes only three ci-operator YAML configuration files. The diff removes or replaces existing CI steps and adds no Ginkgo tests, test bodies, or test wrappers. Therefore, the S…
Topology-Aware Scheduling Compatibility	✅ Passed	PASS. The review range changes only three ci-operator job YAML files. The diff only removes or replaces test-step references; it adds no deployment manifest, operator code, controller, pod affinity, t…
Ote Binary Stdout Contract	✅ Passed	PASS: The pull request changes only three ci-operator YAML files. It selects the existing interop-opp-wait-mcp step and removes or replaces cucushift-installer-check-cluster-health; it adds no OTE…
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS. The authoritative pull-request diff changes only three ci-operator YAML configuration files. It removes or replaces step references and adds no Ginkgo e2e tests, IPv4 logic, IP addresses, or ext…
No-Weak-Crypto	✅ Passed	The pull request changes only CI step references in three YAML files. The diff replaces cucushift-installer-check-cluster-health with interop-opp-wait-mcp; it adds no cryptographic algorithms, cry…
Container-Privileges	✅ Passed	PASS. The pull request changes only test-step references in three ci-operator YAML files. The additions and removals contain none of: privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivile…

Full details: No-Sensitive-Data-In-Logs

Explanation

The PR activates interop-opp-wait-mcp in the OCP 5.0 and 5.1 AWS and vSphere jobs. Its existing command script uses set -x and writes raw cluster diagnostics to job logs, including oc get nodes -o wide at lines 93, 228, and 242, not-ready node names at lines 67-81, and MCP descriptions at lines 243-245. These outputs may expose internal node hostnames and addresses. The 4.22 change leaves an already active wait step in place, but the 5.0 and 5.1 replacements introduce this logging behavior into those jobs. The reviewed diff confirms that the replacement script itself is unchanged; the configuration changes activate it.

Resolution

Sanitize the replacement step before enabling it in these jobs. Disable set -x, avoid logging raw oc get nodes -o wide and oc describe mcp output, and report only aggregate health results or explicitly redacted identifiers. Apply the same logging restriction to all jobs that use interop-opp-wait-mcp.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@redhat-chai-bot: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.0-lpMainline-lp-interop-cr--full-stack--aws	N/A	periodic	Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.0-lpMainline-lp-interop-cr--full-stack--vsphere	N/A	periodic	Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--aws	N/A	periodic	Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--vsphere	N/A	periodic	Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-cr--full-stack--aws	N/A	periodic	Ci-operator config changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

@redhat-chai-bot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[redhat-chai-bot]

/pj-rehearse ack

---

AI-generated. Review for accuracy.

[openshift-merge-bot]

@redhat-chai-bot: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amp-rh, redhat-chai-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/RedHatQE/interop-testing/OWNERS [amp-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[amp-rh]

/retest