[DBMON-5640] Postgres - Compile and reuse regex patterns

[eric-weaver]

What does this PR do?

This PR compiles and reuses regex patterns in the Postgres integration to avoid rebuilding the pattern frequently

Motivation

• PARAMETERIZED_QUERY_PATTERN - This is a fairly expensive regex pattern that does lookaheads and lookbacks of capture groups. We run this static regex pattern in a tight loop for each query sampled that is explained vis parameterized query. This should have the most benefit seen
• Metadata Schema include/exclude patterns - We're currently recompiling the patterns loaded from user configs multiple times each schema collection. This will typically be a small number of includes or exclude patterns. Now we'll compile them on first access and cache it for later use.
• Version parsing patterns - These are called up to once per check run. Minor anticipated benefit from compiling these.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.25%. Comparing base (e4ca807) to head (75cc4c8).
⚠️ Report is 11 commits behind head on master.

Additional details and impacted files

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jasonmp85]

I've always found these comments kinda weird… sure it saves some performance, but only if you explode the Python regex cache to begin with (I think the runtime has one, and I think it can be expanded with a library call)…

If the regex has bad backtracking or something, it being compiled will probably be in the noise…

[eric-weaver]

Yeah this comment is bad and slipped in. Let me remove that so it's not misleading. To clarify the changes, it's less that we'll see improvement in base cases but more that we'll avoid potential performance hit pitfalls on hot loop paths.

As you mentioned below Python does have a lru cache for on the fly regex. It has a a cap of 512 patterns. Way higher than we realistically should see within a single Postgres check instance, but since this runs as a plugin alongside potentially many other instances of this or other checks it's not impossible. We expose config options for passing in lists of regex patterns for customer provided queries in a number integrations which leaves this more susceptible to unknown increases in counts.

The main benefit here is we'll be holding onto a hard reference to the compiled regex directly so this is guaranteeing that our hot path regex don't accidentally get evicted and churned

[jasonmp85]

This sort of seems like the same logic Python already uses internally… https://github.com/python/cpython/blob/fbfbfdf33c3cfe00c6319423ac69f5fb4e877034/Lib/re/__init__.py#L321

[jasonmp85]

This is probably an improvement (is it benchmarked?) but the more of this regex stuff I see in the agent the more I wonder if there isn't a better global approach to this. Python include an internal cache for storing compiled regexes, so calling compile inline is essentially the same as the cache already implemented here, except maybe we don't have enough evidence about how large that cache is for our customers?

It's still a , but I just wanted to raise the question since I've seen this come up several times by now…

Review from jasonmp85 is dismissed. Related teams and files:

• database-monitoring-agent
  • postgres/datadog_checks/postgres/explain_parameterized_queries.py