Skip to content

datadog-agent stop sending metrics #74

@k-cloudit

Description

@k-cloudit

I'm using mimir proxies main branch to send metrics from datadog-agent to mimir.
This this how datadog is configured :

additional_endpoints:
  "http://IP:8009/datadog":
  - grafana-labs

it works, mimir is receiving metrics but after few minutes I have this message in datadog-agent :

2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/check_run": "can't translate series: client_error: client error: 404\n", dropping it
2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/series": "can't translate series: client_error: client error: 404\n", dropping it

and this log message on mimir-proxies :

level=info ts=2023-09-07T11:34:51.057594497Z elapsed=2.481926ms traceID=71652a924ff0db99 sampled=false orgID=fake method=POST uri=/datadog/api/v1/series status=400
level=warn ts=2023-09-07T11:35:06.054259542Z traceID=5e64c8ef0da08f03 request_uri=/datadog/api/v1/check_run method=POST msg="can't translate series" response_code=400 err="client_error: client error: 404"

image
mimir-proxies

I attached a grafana dashboard show how the metrics are ingested, no mimir-proxies or datadog-agent have been restarted.

is there any solution to this behavior ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions