VertexAI Initial draft- prompt response logging

[ishleenk17]

Proposed commit message

Add a new metricset to support prompt and response logging for GCP VertexAI Logging by querying the BigQuery Table

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Issue

• Closes https://github.com/elastic/obs-integration-team/issues/553

Screenshots

TODO:

• 1. Update README
• 2. Check if performance can be improved for data fetch from the BigQuery table.
• 3. Update the data.json
• 4. Check deduplication works with GenerateEventID()
• 5. Finalize the default fetch period
• 6. Add tests

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ishleenk17? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/metricbeat/metricbeat-metricset-gcp-vertexai_logs.md
• docs/reference/metricbeat/exported-fields-gcp.md
• docs/reference/metricbeat/metricbeat-module-gcp.md
• docs/reference/metricbeat/metricbeat-modules.md

[colleenmcginnis]

@ishleenk17 all these changes are expected in the generated docs based on your changes in x-pack/metricbeat/module/gcp/vertexai_logs/_meta/docs.md and x-pack/metricbeat/module/gcp/vertexai_logs/_meta/fields.yml. Some notes below.

[Copilot]

Pull Request Overview

This PR introduces a new metricset called vertexai_logs for the GCP module in Metricbeat to collect Vertex AI prompt-response logs from BigQuery. The metricset enables monitoring and analysis of AI model usage by querying BigQuery tables that contain exported Vertex AI interaction logs.

Key changes include:

• New vertexai_logs metricset implementation with BigQuery integration
• Configuration support for BigQuery table access and credentials
• Comprehensive test coverage and documentation

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
x-pack/metricbeat/module/gcp/vertexai_logs/vertexai_logs.go	Core metricset implementation with BigQuery client and data fetching logic
x-pack/metricbeat/module/gcp/vertexai_logs/data.go	Data structures and event creation logic for Vertex AI log rows
x-pack/metricbeat/module/gcp/vertexai_logs/vertexai_logs_test.go	Comprehensive test suite covering query generation, event creation, and data mapping
x-pack/metricbeat/module/gcp/vertexai_logs/_meta/fields.yml	Field definitions for the vertexai_logs metricset
x-pack/metricbeat/module/gcp/vertexai_logs/_meta/docs.md	Documentation explaining the metricset functionality and configuration
x-pack/metricbeat/module/gcp/vertexai_logs/_meta/data.json	Sample event data for the metricset
Configuration and reference files	Updates to enable and document the new metricset

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}