Skip to content

POC: conditional creation of internal topics in Kafka Connect #20384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: trunk
Choose a base branch
from

Conversation

anton-liauchuk
Copy link
Contributor

This is a small POC to demonstrate the possibility of disabling the creation of internal topics in Kafka Connect.
It might be useful in cases where internal topics contain information about a large number of connectors, and one or more of the original topics were accidentally removed or renamed in the configuration.

The new mode would provide the ability to:

  • block the creation of internal topics.
  • prevent Kafka Connect from starting if required topics are missing.

This way, there is a chance to troubleshoot the root cause instead of unexpected state of all connectors (for example, if the config topic is still present but the offset topic was accidentally removed or renamed, all the connectors will start from scratch).

Out of scope for this POC:

  • no JIRA task yet — this PR is just to gather feedback.
  • not fully implemented — there is a TODO item for the additional logic.
  • the configuration name is not final — proposals are welcome.

@github-actions github-actions bot added triage PRs from the community connect small Small PRs labels Aug 20, 2025
@anton-liauchuk
Copy link
Contributor Author

@C0urante, @mimaison, @chia7712 could you take a look at this POC?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
connect small Small PRs triage PRs from the community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant