-
Notifications
You must be signed in to change notification settings - Fork 216
Description
Improve Logging for Materialized View Read Fallback on Permission Failure
The Problem
Currently, when the connector is used to read a BigQuery Materialized View (MV) and the executing service account lacks the necessary bigquery.tables.getData
permission, the connector's behavior is opaque and leads to a poor user experience.
Instead of failing with a clear permission error, the connector silently fails the direct read from the MV's storage and falls back to re-executing the view's underlying query into a temporary table. This behavior, while a reasonable fallback for standard views, causes significant issues for users of Materialized Views:
- Unexpected Costs: Users are surprised by bills for query executions they did not anticipate, as the fallback is not a free operation.
- Performance Degradation: Re-executing a potentially complex MV query is significantly slower than a direct read from its storage, leading to confusing performance bottlenecks.
- Difficult Troubleshooting: The root cause (a missing IAM permission) is not obvious, leading to lengthy diagnostic cycles. This has been observed in customer support cases and is documented in related issues, such as this Dataform issue where users first encounter materialized views: materialized views create error dataform-co/dataform#1640.
A public issue has been filed against Google Issue Tracker to address the symptoms, but the root cause lies in the connector's logging behavior: https://issuetracker.google.com/issues/296281345.
Desired Behavior
The connector should detect this specific permission failure scenario and log a clear, actionable warning message before initiating the fallback query. This would make the behavior transparent and allow users to immediately understand the root cause and the steps to fix it.
Proposed Fix
The fix is not to change the fallback behavior itself, but to add clear and actionable logging. When the connector attempts a direct read of a Materialized View and receives a permission-denied error from the BigQuery Storage API, it should emit a WARN
level log message.
An ideal log message would be:
WARN: Failed to initiate a direct read from Materialized View 'project.dataset.table' due to a permission error. The service account likely lacks 'bigquery.tables.getData' permission.
Falling back to re-executing the view's underlying query. This will incur additional BigQuery costs and impact performance.
For optimal performance, grant the 'roles/bigquery.dataViewer' role to the principal at the dataset or table level.
This change would greatly improve the supportability of the connector, empower users to self-resolve permission issues quickly, and reduce the diagnostic burden on both customers and support teams.