cmd,collector: optionally disable concurrent scrapes #6
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously approved but merge into the wrong base: #2
Background
The current behavior of the exporter is to open a new database connection on every scrape.
First, here is where the default metrics collector and new-style collectors are registered:
postgres_exporter/cmd/postgres_exporter/main.go
Line 126 in 198454c
postgres_exporter/cmd/postgres_exporter/main.go
Line 143 in 198454c
Prometheus may run these collectors in parallel. Additionally, the
collectors
package is set up to support concurrent scrapes:postgres_exporter/collector/collector.go
Lines 171 to 176 in 198454c
No doubt it's useful for some to have the default and new-style metrics be collected concurrently, and to be able to support concurrent srapes.
Changes
But for PlanetScale, our metric collection system does not scrape the same endpoint concurrently, so concurrent scrapes aren't useful for us. We can also live without whatever time is gained by having default and new-style metrics be collected concurrently: our scrape timeout is 10s, and we expect metrics to be collected much faster than that. If not, then we probably have other problems we need to look at.
Additionally we want to consume as few customer connections as possible. So having the option of using a single, shared connection between default and new-style metrics is good for us. Additionally, if customers have used up all their connections, having each scrape create a new db conn might mean we can't produce metrics. So, having the option to use a single, shared, persistent connection is doubly useful.
Validation
Create a role with a connection limit of 1:
With
--no-concurrent-scrape
Start the exporter with concurrent scraping disabled:
DATA_SOURCE_NAME="postgresql://[email protected]:5432/postgres?sslmode=disable" ./postgres_exporter --no-concurrent-scrape --log.level=info
Scraping metrics shows no errors in output.
With
--concurrent-scrape
Repeat with
--concurrent-scrape
, shows this error:Connection resilience
Verified that
--no-concurrent-scrape
is resilient to Postgres connection resets. After a connection reset, the exporter will log an error:But the connection will be recreated and the scrape (or the next scrape anyway) will succeed. Behavior seems similar to
master
.