Skip to content

Conversation

@dgniewek
Copy link
Collaborator

@dgniewek dgniewek commented Nov 7, 2025

Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility

Changes

dbt Version Support Expansion

  • Expanded all adapter version ranges: >=1.7.x,<2.0.0 (Snowflake, BigQuery, Postgres, Redshift, Glue)
  • Removed dbt-core from base requirements (adapters provide it transitively)
  • Added Snowflake adapter to test matrix (tox.ini)
  • Relaxed Jinja2 constraint: ==3.1.2>=3.1.3,<4

dbt 1.8+ Compatibility Fix

  • Critical: Refactored publish.py to parse manifest.json as plain JSON instead of using dbt Python API
  • Removed imports: dbt.contracts.graph.manifest.Manifest, dbt.contracts.graph.nodes.*
  • Added defensive error handling with .get() for missing keys
  • Works across dbt 1.7.x through 1.10.x (verified with 70 test executions)

Test Coverage

  • Added 17 comprehensive unit tests for manifest parsing:
    • P0 (6 tests): Critical error cases (missing nodes, corrupt JSON, invalid structure)
    • P1 (4 tests): Important scenarios (full metadata, empty columns, multiple models)
    • P2 (7 tests): Edge cases (None values, mixed resource types, empty manifests)
  • All tests use plain JSON dicts instead of dbt Python objects

Documentation

  • Updated coverage badge (CodeClimate → shields.io, 95%)
  • Added troubleshooting section for dbt pre-release edge case
  • Created design doc: design/001-dbt-manifest-api-migration.md (complete ADR with test matrix)
  • Updated AGENTS.md with dbt 1.8+ fix details and PEP 440 pre-release behavior

Dependency Cleanup

  • Removed MarkupSafe pin (managed by Jinja2)
  • Removed Werkzeug (unused)
  • Fixed GitPython optional import

Breaking Changes

None. Users must continue installing with adapter extras:

pip install data-pipelines-cli[snowflake]  # Required (no change)

@dgniewek dgniewek requested review from kosmag and p-pekala November 7, 2025 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants