Skip to content

kafka: document ${partition} variable #215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 28, 2025
Merged

kafka: document ${partition} variable #215

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5ed9cee
document ${partition} variable
jerrinot Jul 28, 2025
f278d35
tweaks
jerrinot Jul 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions documentation/third-party-tools/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,9 +287,9 @@ Example: Consider the following Kafka message:

The connector will create a table with the following columns:

| firstname <sub>string</sub> | lastname <sub>string</sub> | age <sub>long</sub> | address_street <sub>string</sub> | address_city <sub>string</sub> |
| --------------------------- | -------------------------- | ------------------- | -------------------------------- | ------------------------------ |
| John | Doe | 30 | Main Street | New York |
| firstname <sub>varchar</sub> | lastname <sub>varchar</sub> | age <sub>long</sub> | address_street <sub>varchar</sub> | address_city <sub>varchar</sub> |
|------------------------------|-----------------------------|---------------------|-----------------------------------|---------------------------------|
| John | Doe | 30 | Main Street | New York |

### Client configuration string

Expand Down Expand Up @@ -568,9 +568,10 @@ table=example_table
[...]
```

The `table` options supports simple templating. You can use the following
variables in the table name: `${topic}`, `${key}`. The connector will replace
these variables with the actual topic name and key value from the Kafka message.
The `table` option supports simple templating. You can use the following
variables in the table name: `${topic}`, `${key}`, `${partition}`. The connector will replace
these variables with the actual topic name, key value, and partition number from the Kafka message.


Example:

Expand All @@ -579,14 +580,15 @@ name=questdb-sink
connector.class=io.questdb.kafka.QuestDBSinkConnector
client.conf.string=http::addr=localhost:9000;
topics=example-topic
table=from_kafka_${topic}_${key}_suffix
table=from_kafka_${topic}_${key}_${partition}_suffix

[...]
```

The placeholder `${key}` will be replaced with the actual key value from the
Kafka message. If the key is not present in the message, the placeholder will be
replaced with the string `null`.
replaced with the string `null`. The placeholder `${partition}` will be replaced
with the Kafka partition number from which the message originates.

#### Table schema

Expand Down