[outputs.loki] Colons are not sanitized #17919
Description
Relevant telegraf.conf
[[inputs.file]]
files = ["/etc/telegraf/metrics.out"]
data_format = "influx"
[[outputs.loki]]
domain = "http://loki:3100"
sanitize_label_names = trueLogs from Telegraf
2025-11-01T07:26:42Z I! Loading config: /etc/telegraf/telegraf.conf
2025-11-01T07:26:42Z I! Starting Telegraf 1.36.3 brought to you by InfluxData the makers of InfluxDB
2025-11-01T07:26:42Z I! Available plugins: 239 inputs, 9 aggregators, 35 processors, 26 parsers, 65 outputs, 6 secret-stores
2025-11-01T07:26:42Z I! Loaded inputs: file
2025-11-01T07:26:42Z I! Loaded aggregators:
2025-11-01T07:26:42Z I! Loaded processors:
2025-11-01T07:26:42Z I! Loaded secretstores:
2025-11-01T07:26:42Z I! Loaded outputs: loki
2025-11-01T07:26:42Z I! Tags enabled: host=3d9f2e38e971
2025-11-01T07:26:42Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"3d9f2e38e971", Flush Interval:10s
2025-11-01T07:26:42Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-11-01T07:26:42Z D! [agent] Initializing plugins
2025-11-01T07:26:42Z D! [agent] Connecting outputs
2025-11-01T07:26:42Z D! [agent] Attempting connection to [outputs.loki]
2025-11-01T07:26:42Z D! [agent] Successfully connected to outputs.loki
2025-11-01T07:26:42Z D! [agent] Starting service inputs
2025-11-01T07:26:52Z D! [outputs.loki] Buffer fullness: 1 / 10000 metrics
2025-11-01T07:26:52Z E! [agent] Error writing to outputs.loki: when writing to [http://loki:3100/loki/api/v1/push] received status code, 400: couldn't parse labels: 1:50: parse error: unexpected character inside braces: ':'
2025-11-01T07:26:54Z D! [agent] Stopping service inputs
2025-11-01T07:26:54Z D! [agent] Input channel closed
2025-11-01T07:26:54Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-11-01T07:26:54Z D! [outputs.loki] Buffer fullness: 1 / 10000 metrics
2025-11-01T07:26:54Z E! [agent] Error writing to outputs.loki: when writing to [http://loki:3100/loki/api/v1/push] received status code, 400: couldn't parse labels: 1:50: parse error: unexpected character inside braces: ':'
2025-11-01T07:26:54Z I! [agent] Stopping running outputs
2025-11-01T07:26:54Z D! [agent] Stopped Successfully
System info
Telegraf 1.36.3
Docker
services:
telegraf:
image: telegraf:latest
volumes:
- ./telegraf:/etc/telegraf:ro
command: ["telegraf", "--config", "/etc/telegraf/telegraf.conf", "--debug"]
loki:
image: grafana/loki:3.5.7
ports:
- "3100:3100"Steps to reproduce
- mount this file to
/etc/telegraf/metrics.outin telegraf container
myMeasurement,tag:withcolon1=value1 fieldKey="fieldValue" 1761981698181582000
- replace time stamp with
date +%s%Nif it's too old
docker compose up
Expected behavior
Loki accepts this request
Actual behavior
reproduce-telegraf-loki-integ-telegraf-1 | 2025-11-01T07:26:52Z E! [agent] Error writing to outputs.loki: when writing to [http://loki:3100/loki/api/v1/push] received status code, 400: couldn't parse labels: 1:50: parse error: unexpected character inside braces: ':'
reproduce-telegraf-loki-integ-loki-1 | level=error ts=2025-11-01T07:26:52.082286541Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="couldn't parse push request: couldn't parse labels: 1:50: parse error: unexpected character inside braces: ':'" org_id=fake
Additional info
This seems to be a documentation issue in loki; the doc says it supports colons but it actually not: grafana/loki#19673