Skip to content

[Feature Request] Improve Logging for Materialized View Read Fallback on Permission Failure #1401

@cjac

Description

@cjac

Improve Logging for Materialized View Read Fallback on Permission Failure

The Problem

Currently, when the connector is used to read a BigQuery Materialized View (MV) and the executing service account lacks the necessary bigquery.tables.getData permission, the connector's behavior is opaque and leads to a poor user experience.

Instead of failing with a clear permission error, the connector silently fails the direct read from the MV's storage and falls back to re-executing the view's underlying query into a temporary table. This behavior, while a reasonable fallback for standard views, causes significant issues for users of Materialized Views:

  • Unexpected Costs: Users are surprised by bills for query executions they did not anticipate, as the fallback is not a free operation.
  • Performance Degradation: Re-executing a potentially complex MV query is significantly slower than a direct read from its storage, leading to confusing performance bottlenecks.
  • Difficult Troubleshooting: The root cause (a missing IAM permission) is not obvious, leading to lengthy diagnostic cycles. This has been observed in customer support cases and is documented in related issues, such as this Dataform issue where users first encounter materialized views: materialized views create error dataform-co/dataform#1640.

A public issue has been filed against Google Issue Tracker to address the symptoms, but the root cause lies in the connector's logging behavior: https://issuetracker.google.com/issues/296281345.

Desired Behavior

The connector should detect this specific permission failure scenario and log a clear, actionable warning message before initiating the fallback query. This would make the behavior transparent and allow users to immediately understand the root cause and the steps to fix it.

Proposed Fix

The fix is not to change the fallback behavior itself, but to add clear and actionable logging. When the connector attempts a direct read of a Materialized View and receives a permission-denied error from the BigQuery Storage API, it should emit a WARN level log message.

An ideal log message would be:

WARN: Failed to initiate a direct read from Materialized View 'project.dataset.table' due to a permission error. The service account likely lacks 'bigquery.tables.getData' permission.
Falling back to re-executing the view's underlying query. This will incur additional BigQuery costs and impact performance.
For optimal performance, grant the 'roles/bigquery.dataViewer' role to the principal at the dataset or table level.

This change would greatly improve the supportability of the connector, empower users to self-resolve permission issues quickly, and reduce the diagnostic burden on both customers and support teams.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions