Repository navigation
Fix redpanda default log_retention_ms=-1 using Job - #89
Merged
Merged
Conversation
bootstrap.yaml's log_retention_ms: -1 (0fbe8e6) does not survive a restart on redpanda v24.2. At first start the bootstrap import serializes the cluster property through an unsigned 64 bit integer, so the controller log and config_cache.yaml get 18446744073709551615. Redpanda rejects that value: `rpk cluster config status` lists log_retention_ms as INVALID, and at every restart the broker logs "Ignoring invalid property" and uses the 7 day default. Any negative value takes the same path. So a new cluster keeps records only until its first restart. The value is now set through the admin API, which stores it correctly, by the Job redpanda-cluster-config in kafka/base. It follows the redpanda chart's post-upgrade job (the chart's hooks don't run under kustomize, and its 5.7 job uses `config set key -1`, which rpk reads as a flag): the redpanda image's rpk, with the chart's redpanda.yaml for the admin API address. It waits for the admin API, sets each key=value from its args that differs, then checks `config get` and that `config status` lists nothing INVALID, and fails otherwise. The args are the one place for such properties, one line each. The Job carries yolean.se/converge-mode: replace, as kafka/topic-job does, so yconverge deletes and re-creates it at every converge. That handles the immutable pod template when the args change, and re-applies the declared values if a cluster lost them. A rerun with nothing to change leaves the config version as it is. A plain `kubectl apply -k` leaves an unchanged Job alone, and rejects a changed one as immutable rather than skipping it. No ttlSecondsAfterFinished: the converge replaces the Job anyway, the completed Job keeps the log of the last run, and the new check in k3s/40-kafka (wait for condition=complete) can't race a deletion. Verified with yconverge -k k3s/40-kafka on a throwaway cluster with ghcr.io/yolean/redpanda:v24.2.14: - fresh converge: the Job sets -1, `config get` prints -9223372036854775808 (rpk's display of infinite), status lists nothing INVALID, a topic shows retention.ms -1 DEFAULT_CONFIG; - after a broker restart: the same, no "Ignoring invalid property"; - converge again unchanged: the Job reruns, sets nothing, config version stays 3; - args changed to 86400000: the Job is re-created and sets it (version 4), and back to -1 (version 5); no broker restart or StatefulSet revision change; - a cluster converged from main and restarted (7 days, INVALID), then converged with this change: -1, nothing INVALID, also after a restart; - an invalid value or an arg without = fails the Job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to #88, inspired by later generations of the upstream helm chart, which we want to avoid due to having an operator