Repository navigation
Fix two kuttl test races: multi-pod log read and pgbouncer certificate rotation - #4573
Open
ValClarkson wants to merge 3 commits into
Conversation
The replica-create backup Job is meant to fail, so Kubernetes replaces its Pod and several Pods carry the pgbackrest-backup=replica-create label by the time step 01 runs. `kubectl get pods -o name` returned all of them as one newline-joined string, and passing that to `kubectl logs` failed with "arguments in resource/name form may not have more than one slash". The log read came back empty and the step reported "did not find expected standby cluster error", which had nothing to do with standby clusters. Some of those Pods also exit on the pgBackRest configuration hash comparison before a backup runs, so their logs cannot contain the expected error even when nothing is wrong. Iterate over every matching Pod, retry the lookup and the log read together rather than reading once, and print each Pod's exit code and logs when the error is not found.
Step 12 wiped the stored PgBouncer certificate and then treated the first 'Loaded' line from the pgbouncer-config sidecar as proof that the regenerated certificate was live. That line can be the reload caused by the wipe itself: PGO regenerates the certificate within a couple of seconds, but kubelet needs considerably longer to propagate the updated Secret into the pod's projected volume. When the wipe's own reload was matched, step 13 read back the unchanged certificate and step 14 failed with 'Certificate did not change!'. Wait instead for the certificate mounted in the pod to differ from the one served before the wipe. That is the condition that decides what step 13 reads, so matching it removes the race rather than narrowing it.
kuttl runs a TestStep's commands exactly once and only loops the assert phase, so the retry helper in this file never retried; it printed a message, slept five seconds and failed the step. Every other file in the tree that uses that helper is a TestAssert. Make this one an assert too so kuttl owns the retrying, and drop the bounded shell loop that was standing in for it. Nesting a shell loop inside a command kuttl already re-runs multiplies the two timeouts and repeats the failure output on every attempt. The per-pod iteration stays, since a single pass still has to check every Pod. Move the verbose log dump into a collector, which kuttl runs once after the assert gives up rather than on each attempt.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #4570 to REL_5_8.
pgbackrest-backup-standby:kubectl get pods -o namereturns one line perpod, which fails
kubectl logswith "arguments in resource/name form may nothave more than one slash". Select a single pod and move the check into
01-assert.yamlso kuttl retries it.pgbouncer: step 12 accepted the firstLoadedline in the log, which wasusually its own wipe's reload rather than the rotation. Wait for the
regenerated certificate.
Test-only; no product code changes.