-
Notifications
You must be signed in to change notification settings - Fork 5k
VertexAI Initial draft- prompt response logging #46383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ishleenk17
merged 31 commits into
elastic:main
from
ishleenk17:ish_gcp_vertexai_prompt_response
Sep 16, 2025
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
629c206
VertexAI Initial draft- prompt response logging
ishleenk17 53f0461
Add changelog
ishleenk17 21366f7
make update
ishleenk17 0e8f5c8
Add test file
ishleenk17 ff778fd
Add documentation
ishleenk17 124e284
Update docs
ishleenk17 e7a3c9a
Update documentaion and make update
ishleenk17 cf6b361
Resolve lint errors
ishleenk17 62e8541
Merge branch 'main' into ish_gcp_vertexai_prompt_response
ishleenk17 11c9010
Updated test file
ishleenk17 c06a75d
Remove test file
ishleenk17 6f3d5d0
Add test file
ishleenk17 09bc676
update test file
ishleenk17 d050a1f
Update x-pack/metricbeat/module/gcp/vertexai_logs/_meta/docs.md
ishleenk17 162acd4
Merge branch 'main' into ish_gcp_vertexai_prompt_response
ishleenk17 f005382
Update the partiotin logic in BigQuery
ishleenk17 d72b583
Add support of logging time filtering
ishleenk17 475cf89
reshuffle imports in vertexai_logs_test.go
mykola-elastic 44a151a
Add string datatype for request id
ishleenk17 27aea86
Merge branch 'main' into ish_gcp_vertexai_prompt_response
ishleenk17 1fcbea2
Merge branch 'main' into ish_gcp_vertexai_prompt_response
ishleenk17 f9b152b
Address comments
ishleenk17 336920e
Remove the otel_log field
ishleenk17 cecdb66
Update vertexai_logs.go
ishleenk17 02537e9
Update vertexai_logs_test.go
ishleenk17 c3bd7a9
Replace run with read
ishleenk17 9adad7b
Linter error
ishleenk17 7f51005
linter fix
ishleenk17 ee38195
LINTER ERROR
ishleenk17 dcd078f
Merge branch 'main' into ish_gcp_vertexai_prompt_response
colleenmcginnis 14f1834
rerun make update
colleenmcginnis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
184 changes: 184 additions & 0 deletions
184
docs/reference/metricbeat/metricbeat-metricset-gcp-vertexai_logs.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| --- | ||
| mapped_pages: | ||
| - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-gcp-vertexai_logs.html | ||
| applies_to: | ||
| stack: beta 9.2.0 | ||
| --- | ||
|
|
||
| % This file is generated! See scripts/docs_collector.py | ||
|
|
||
| # Google Cloud Platform vertexai_logs metricset [metricbeat-metricset-gcp-vertexai_logs] | ||
|
|
||
| The `vertexai_logs` metricset is designed to collect Vertex AI prompt-response logs from GCP BigQuery. BigQuery is a fully-managed, serverless data warehouse that stores detailed logs of interactions with Vertex AI models. | ||
|
|
||
| Vertex AI logs export to BigQuery enables you to export detailed Google Cloud Vertex AI interaction data (such as prompts, responses, model usage, and metadata) automatically to a BigQuery dataset that you specify. Then you can access your Vertex AI logs from BigQuery for detailed analysis and monitoring using Metricbeat. This enables comprehensive tracking of AI model usage, performance monitoring, and cost analysis. | ||
|
|
||
| The logs include detailed information about: | ||
| - API endpoints and deployed models | ||
| - Request and response payloads | ||
| - Model versions and API methods used | ||
| - Request metadata and timing information | ||
|
|
||
|
|
||
| ## Metricset-specific configuration notes [_metricset_specific_configuration_notes_14] | ||
|
|
||
| * **table_id**: (Required) Full table identifier in the format `project_id.dataset_id.table_name` that contains the Vertex AI logs data. You can copy this from the "Details" tab when viewing your table in the BigQuery web console, under the "Table ID" field. | ||
|
|
||
|
|
||
| ## Configuration example [_configuration_example_22] | ||
|
|
||
| ```yaml | ||
| - module: gcp | ||
| metricsets: | ||
| - vertexai_logs | ||
| period: 10m | ||
| project_id: "your project id" | ||
| credentials_file_path: "your JSON credentials file path" | ||
| table_id: "your_project.your_dataset.your_vertex_ai_logs_table" | ||
| ``` | ||
|
|
||
| ## Sample Event | ||
|
|
||
| Here is a sample event for `vertexai_logs`: | ||
|
|
||
| ```json | ||
| { | ||
| "@timestamp": "2023-12-01T10:30:45.000Z", | ||
| "cloud": { | ||
| "provider": "gcp", | ||
| "project": { | ||
| "id": "my-gcp-project" | ||
| } | ||
| }, | ||
| "gcp": { | ||
| "vertexai_logs": { | ||
| "endpoint": "https://us-central1-aiplatform.googleapis.com", | ||
| "deployed_model_id": "1234567890123456789", | ||
| "logging_time": "2023-12-01T10:30:45.000Z", | ||
| "request_id": 98765432101234567, | ||
| "request_payload": ["What is machine learning?"], | ||
| "response_payload": ["Machine learning is a subset of artificial intelligence..."], | ||
| "model": "gemini-2.5-pro", | ||
| "model_version": "1.0", | ||
| "api_method": "generateContent", | ||
| "full_request": { | ||
| "inputs": ["What is machine learning?"], | ||
| "parameters": { | ||
| "temperature": 0.7 | ||
| } | ||
| }, | ||
| "full_response": { | ||
| "outputs": ["Machine learning is a subset of artificial intelligence..."], | ||
| "usage": { | ||
| "input_tokens": 5, | ||
| "output_tokens": 50 | ||
| } | ||
| }, | ||
| "metadata": { | ||
| "user_id": "user123", | ||
| "session_id": "session456" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Fields [_fields] | ||
|
|
||
| For a description of each field in the metricset, see the [exported fields](/reference/metricbeat/exported-fields-gcp.md) section. | ||
|
|
||
| Here is an example document generated by this metricset: | ||
|
|
||
| ```json | ||
| { | ||
| "@timestamp": "2025-09-02T10:14:50.313Z", | ||
| "agent": { | ||
| "hostname": "metricbeat-host", | ||
| "name": "metricbeat-host" | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "elastic-beats" | ||
| }, | ||
| "provider": "gcp", | ||
| "project": { | ||
| "id": "elastic-beats" | ||
| } | ||
| }, | ||
| "event": { | ||
| "dataset": "gcp.vertexai_logs", | ||
| "duration": 123456789, | ||
| "module": "gcp" | ||
| }, | ||
| "gcp": { | ||
| "vertexai_logs": { | ||
| "endpoint": "https://us-central1-aiplatform.googleapis.com/v1/projects/elastic-beats/locations/us-central1/endpoints/123456789", | ||
| "deployed_model_id": "model-deployment-123", | ||
| "logging_time": "2025-09-02T10:14:50.313Z", | ||
| "request_id": 98765432101234567, | ||
| "model": "gemini-2.5-pro", | ||
| "model_version": "001", | ||
| "api_method": "generateContent", | ||
| "request_payload": [ | ||
| "What is the weather like today?" | ||
| ], | ||
| "response_payload": [ | ||
| "I don't have access to real-time weather information. Please check a weather service or app for current conditions." | ||
| ], | ||
| "full_request": { | ||
| "contents": [ | ||
| { | ||
| "parts": [ | ||
| { | ||
| "text": "What is the weather like today?" | ||
| } | ||
| ], | ||
| "role": "user" | ||
| } | ||
| ], | ||
| "generationConfig": { | ||
| "temperature": 0.7, | ||
| "maxOutputTokens": 1024 | ||
| } | ||
| }, | ||
| "full_response": { | ||
| "candidates": [ | ||
| { | ||
| "content": { | ||
| "parts": [ | ||
| { | ||
| "text": "I don't have access to real-time weather information. Please check a weather service or app for current conditions." | ||
| } | ||
| ], | ||
| "role": "model" | ||
| }, | ||
| "finishReason": "STOP", | ||
| "safetyRatings": [ | ||
| { | ||
| "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", | ||
| "probability": "NEGLIGIBLE" | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "usageMetadata": { | ||
| "promptTokenCount": 8, | ||
| "candidatesTokenCount": 24, | ||
| "totalTokenCount": 32 | ||
| } | ||
| }, | ||
| "metadata": { | ||
| "region": "us-central1", | ||
| "zone": "us-central1-a" | ||
| } | ||
| } | ||
| }, | ||
| "metricset": { | ||
| "name": "vertexai_logs", | ||
| "period": 300000 | ||
| }, | ||
| "service": { | ||
| "type": "gcp" | ||
| } | ||
| } | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.