-
Notifications
You must be signed in to change notification settings - Fork 7
Description
I'm using mimir proxies main branch to send metrics from datadog-agent to mimir.
This this how datadog is configured :
additional_endpoints:
"http://IP:8009/datadog":
- grafana-labs
it works, mimir is receiving metrics but after few minutes I have this message in datadog-agent :
2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/check_run": "can't translate series: client_error: client error: 404\n", dropping it
2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/series": "can't translate series: client_error: client error: 404\n", dropping it
and this log message on mimir-proxies :
level=info ts=2023-09-07T11:34:51.057594497Z elapsed=2.481926ms traceID=71652a924ff0db99 sampled=false orgID=fake method=POST uri=/datadog/api/v1/series status=400
level=warn ts=2023-09-07T11:35:06.054259542Z traceID=5e64c8ef0da08f03 request_uri=/datadog/api/v1/check_run method=POST msg="can't translate series" response_code=400 err="client_error: client error: 404"


I attached a grafana dashboard show how the metrics are ingested, no mimir-proxies or datadog-agent have been restarted.
is there any solution to this behavior ?