Ingest internal telemetry from the OTel Collector when it is running

[faec]

Add a monitoring component to ingest telemetry values from the collector's prometheus endpoint into the elastic_agent.collector dataset.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

Enable agent monitoring while at least one component (including monitoring itself) uses the otel runtime. Prometheus metrics from the collector should be ingested to the elastic_agent.collector dataset.

Related issues

• Closes [beats receivers] OTel based pipeline must continue to report queue metrics for the elastic_agent integration #8032

[mergify]

This pull request does not have a backport label. Could you fix it @faec? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[faec]

This was the least verbose way I could find to extract the prometheus endpoint configuration (the mapstructure annotations don't seem to accept dotted field names), but I'd be happy to learn if there's a better one.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: a05982e

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11
• Win2022:sudo:default
• Win2022:sudo:default
• Win2022:sudo:fleet
• Win2022:non-sudo:default
• Win2022:non-sudo:default
• Win2025:sudo:default
• Win2025:sudo:default
• Win2025:non-sudo:default
• Win2025:non-sudo:default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:sudo: default
• x86_64:sudo: default
• arm:sudo: default
• arm:sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:sudo: default
• x86_64:sudo: default

History

cc @faec

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate failed

Failed conditions
2.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube