feat: Implement provider configuration system with full referenceability #53

eveld · 2025-07-14T07:12:37Z

Provider Configuration with DAG Integration

Summary

Major architectural change: Moved providers into the DAG system, enabling provider configurations to reference values from resources and variables
Provider lifecycle management: Providers are now initialized during DAG execution with full dependency resolution
Enhanced referenceability: Provider config fields are now referenceable in HCL expressions (e.g., provider.database.config.value)
Test isolation improvements: Fixed all tests to use proper test isolation patterns and removed unnecessary duration testing
Comprehensive test coverage: Added extensive tests for provider-resource dependencies and DAG integration

Key Changes

Architectural Transformation

Providers in DAG: Removed provider exclusion from DAG processing, making them full participants in dependency resolution
Automatic dependencies: Resources automatically depend on their providers, ensuring proper execution order
Deferred initialization: Provider initialization moved from parsing time to DAG execution time for proper context resolution
Enhanced referenceability: Early resolution of simple provider configs during parsing for better HCL referenceability

Provider Lifecycle Management

New initializeProvider(): Handles provider initialization during DAG execution with full evaluation context
New getResourceProviderName(): Extracts provider names from resources using reflection for explicit and implicit providers
New updateProviderContextVariable(): Updates provider context variables with resolved config values
Enhanced callProviderLifecycle(): Now handles both provider initialization and resource lifecycle operations

DAG Integration

Provider dependencies: Added automatic provider dependency logic in doYaLikeDAGs()
Special provider handling: Modified walkCallback() to handle providers with global context access
Resource-provider linking: Resources now properly depend on their providers in the dependency graph
Module support: Provider dependencies work correctly across module boundaries

Parser Improvements

Deferred registration: Providers are stored but not fully initialized during parsing
Early config resolution: Simple configs (variables/literals) are resolved during parsing for referenceability
Duplicate detection: Improved provider name validation without premature initialization
Context management: Better separation between parsing context and execution context

Testing Improvements

Test isolation: Fixed all tests to use setupParser(t) instead of manual NewParser() construction
Removed duration testing: Eliminated unnecessary and unreliable duration assertions from event tests
Removed production sleeps: Removed artificial delays that were added solely for test compatibility
Provider dependency tests: Added comprehensive tests in dag_test.go for provider-resource dependencies
Error scenario testing: Enhanced error callback testing with proper state setup

Key Features Enabled

Variable-driven providers: Provider configs can reference variables (e.g., provider.db { host = variable.db_host })
Resource-driven providers: Provider configs can reference resource outputs (e.g., certificates from other resources)
Config referenceability: Provider config fields accessible in expressions (e.g., provider.k8s.config.endpoint)
Terraform-like enforcement: Resources MUST have corresponding provider blocks defined
Full lifecycle support: Providers support init, create, refresh, update, destroy operations

Advanced Use Cases

This implementation enables powerful patterns like:

resource "tls_cert" "ca" {
  # Generate a CA certificate
}

provider "kubernetes" {
  config {
    ca_certificate = resource.tls_cert.ca.cert_pem
    endpoint = "https://k8s.example.com"
  }
}

resource "k8s_deployment" "app" {
  provider = "kubernetes"
  # This deployment uses the dynamically configured k8s provider
}

Integration

Successfully builds on existing provider configuration branch
All existing tests pass with no regressions
Maintains backward compatibility for simple provider configs
Enhanced error handling and validation
Proper event firing for all provider operations

Test Plan

All unit tests pass (go test -v)
Provider dependency tests pass
DAG integration tests pass
Event callback tests pass (without duration requirements)
Error scenario tests pass
No import cycle issues
Module-scoped provider dependencies work
Resource-provider enforcement works
Provider config referenceability works

This represents a significant architectural advancement that brings provider configuration capabilities much closer to Terraform's model while maintaining the flexibility and power of the existing HCL configuration system.

This commit implements a comprehensive provider configuration system that makes providers referenceable like variables using the syntax provider.name.config.property. ## Core Provider Implementation: - **Provider resource type**: New Provider struct extending ResourceBase for FQRN support - **Three-phase parsing**: Variables → Providers → Resources for proper dependency resolution - **Provider blocks**: Full HCL support for provider configuration with dynamic config blocks - **Plugin integration**: Automatic config type detection and validation through plugin registry - **FQRN referenceability**: Complete support for provider.name.config.property syntax ## Provider Infrastructure: - Added Provider to builtin resource types and FQRN parsing - Implemented RegisterProvider/GetProvider/ListProviders in PluginRegistry - Provider-specific parsing with manual cty object reconstruction for config fields - DAG exclusions for providers (like variables - referenceable but no lifecycle processing) - Comprehensive error handling with detailed validation messages ## Plugin System Enhancements: - **Automatic GetConfigType()**: PluginBase now provides config type detection automatically - **Enhanced RegisterResourceProvider**: Updated to capture config types during registration - **Provider source mapping**: RegisterPluginSource for connecting HCL sources to plugins - **Type-safe config conversion**: HCL body → concrete config type conversion with validation ## Example Implementation: - **ContainerdPlugin**: Complete example provider with socket/namespace/runtime configuration - **Provider example HCL**: Demonstrates variable interpolation in provider config - **Container resource**: Shows provider usage with explicit provider field - **Integration tests**: End-to-end testing of provider parsing and referenceability ## Parsing Enhancements: - **parseProvidersInFile**: Dedicated provider parsing phase after variables - **parseProviderResource**: Provider-specific parsing with config block handling - **Manual cty reconstruction**: Ensures provider.config.* references work correctly - **Context integration**: Providers available in HCL evaluation context for interpolation ## Code Quality & Cleanup: - Consistent API patterns (removed redundant wrapper functions) - Updated all plugins to use automatic GetConfigType() from PluginBase - Comprehensive test coverage for provider parsing, validation, and error scenarios - Clean separation between parser logic and plugin registry responsibilities ## Technical Details: - Provider config stored as interface{} for plugin flexibility, converted to concrete types - Full validation of provider sources, required fields, and duplicate detection - Provider blocks parsed separately from resources to enable provider-dependent resources - Support for provider inheritance and default provider resolution This implementation enables clean, type-safe provider configurations while maintaining full HCL expressiveness and providing comprehensive error reporting.

Resolved conflicts by: - Integrating new event callback functionality and destroy methods from v2 - Maintaining provider configuration system implementation - Updating TestResourceProvider initialization to include {plugin: p} parameter - Keeping provider config variable for our implementation - Preserving both provider parsing tests and new v2 test functionality This merge brings together the provider configuration feature with the latest v2 enhancements including event callbacks, destroy lifecycle, and other improvements.

Updated GetProvider(resource) to GetProviderAdapter(resource) to match the correct method signature for getting provider adapters during resource lifecycle operations.

- Fix provider exclusion in destroyWalkCallback to prevent "no provider found" errors - Replace all NewParser(nil) calls with setupParser(t) for proper test isolation - Ensure TestParsingOrderVariableOverrideOrder uses setupParser with custom options - All provider configuration and v2 event callback tests now pass The v2 merge is now complete with proper test isolation and no state persistence issues between tests.

…olation ARCHITECTURAL CHANGES: - Move providers into the DAG: Providers are now full participants in dependency resolution, allowing provider configurations to reference values from resources and variables - Provider lifecycle management: Providers are initialized during DAG execution rather than immediately during parsing, enabling proper dependency resolution - Automatic provider dependencies: Resources automatically depend on their providers in the DAG, ensuring providers are processed before dependent resources IMPLEMENTATION: - Add getResourceProviderName() to extract provider names from resources using reflection - Add initializeProvider() for execution-time provider initialization with full context - Add updateProviderContextVariable() to make provider config fields referenceable - Modify walkCallback() to handle providers specially with global context access - Update parseProviderResource() to defer initialization and enable early referenceability - Remove explicit provider exclusion from DAG processing TESTING IMPROVEMENTS: - Fix test isolation: Replace manual NewParser() calls with setupParser(t) across all tests - Remove unnecessary duration testing from event callbacks (focus on meaningful assertions) - Remove production code sleep that was added solely for test compatibility - Add comprehensive provider dependency tests in dag_test.go - Move provider dependency tests to follow Go naming conventions KEY FEATURES: - Provider configs can now reference variables and resource outputs - Provider config fields are referenceable (e.g., provider.database.config.value) - Resources must have corresponding provider blocks (enforced like Terraform) - Full provider lifecycle support: init, create, refresh, update, destroy - Proper error handling and event firing for all provider operations This enables advanced use cases like configuring a Kubernetes provider with certificates generated by other resources, making the system much more flexible and powerful while maintaining proper dependency ordering.

nicholasjackson · 2025-07-15T07:50:05Z

dag.go

+				// Store the concrete config
+				provider.Config = configPtr
+			} else {
+				// If no context available, leave config as hcl.Body for now


I am not sure this is valid at this point, when parsing a provider you should always have context.

nicholasjackson · 2025-07-15T07:51:55Z

dag.go

+
+	// Convert hcl.Body config to concrete type if config is provided
+	if provider.Config != nil {
+		if configBody, ok := provider.Config.(hcl.Body); ok {


This cast looks like it silently fails, we should panic if this does not succede as it should not be possible that this can fail

nicholasjackson · 2025-07-15T07:59:11Z

dag_test.go

+	testFile := filepath.Join(tmpDir, "test.hcl")
+
+	hclContent := `
+variable "test_value" {


For these type of tests we normally create test fixtures rather than hard coding. Test fixtures are also really useful documentation.

nicholasjackson · 2025-07-15T08:09:02Z

internal/resources/provider.go

+	Config interface{} `json:"config,omitempty"`
+
+	// Internal fields for plugin management (not exposed via HCL)
+	Plugin       plugins.Plugin `json:"-"`


I think this causes a circular reference we should avoid the provider having access to its parent

nicholasjackson · 2025-07-15T08:40:40Z

parser.go

+				if configType != nil {
+					// Try to decode config with current context (variables should be available)
+					configPtr := reflect.New(configType).Interface()
+					diags := gohcl.DecodeBody(configBody, ctx, configPtr)


The context is almost always going to be not set here and this will fail, full decode should only happen in the dag

nicholasjackson · 2025-07-15T08:42:33Z

parser.go

 	// lazy process on dag walk
-	if b.Type == string(resources.TypeVariable) {
-		diag := gohcl.DecodeBody(b.Body, ctx, p)
+	if b.Type == string(resources.TypeVariable) || b.Type == string(resources.TypeProvider) {


I think it is better to treat a provider exactly like a resource, if we handle it at the same time all the dependencies will also be calculated for the dag.

nicholasjackson · 2025-07-15T08:43:18Z

parser_test.go

+	testFile := filepath.Join(tmpDir, "test.hcl")
+
+	hclContent := `
+# Variable defined after provider - but should still work due to three-phase parsing


These tests should use the convention of test fixtures

nicholasjackson · 2025-07-15T08:44:36Z

plugin_registry.go

+	// Plugin and provider management
+	// plugins: maps source identifiers (e.g., "jumppad/containerd") to plugin implementations
+	// This allows finding which plugin handles a given source when registering providers
+	plugins      map[string]plugins.Plugin  // source -> plugin mapping


I don't really think this needs to change

nicholasjackson · 2025-07-15T08:45:36Z

plugin_registry.go

+
+// RegisterProvider registers a provider instance with its configuration
+// This is called after the provider resource has been parsed to set up plugin-specific fields
+func (r *PluginRegistry) RegisterProvider(provider *resources.Provider, ctx *hcl.EvalContext) error {


A provider should just be part of a plugin, you should not need to register it separately

- Remove else clause in dag.go for provider context handling - Add panic for config body cast failures instead of silent failure - Convert hardcoded HCL strings to organized test fixtures - Remove Plugin field from Provider struct to avoid circular references - Remove early provider config decoding to fix dependency resolution - Treat providers like resources in dependency handling - Update comment in utils_test.go to specify exact fixture path All tests pass with 60s timeout. Fixes issues #1-6 from PR review.

eveld requested a review from nicholasjackson July 14, 2025 07:12

eveld added 4 commits July 14, 2025 09:32

Fix method call in DAG processing after v2 merge

100ed90

Updated GetProvider(resource) to GetProviderAdapter(resource) to match the correct method signature for getting provider adapters during resource lifecycle operations.

eveld changed the title ~~Implement provider configuration system with full referenceability~~ feat: Implement provider configuration system with full referenceability Jul 15, 2025

nicholasjackson reviewed Jul 15, 2025

View reviewed changes

eveld marked this pull request as draft July 16, 2025 07:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Implement provider configuration system with full referenceability #53

feat: Implement provider configuration system with full referenceability #53

Uh oh!

eveld commented Jul 14, 2025 •

edited

Loading

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

nicholasjackson Jul 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: Implement provider configuration system with full referenceability #53

Are you sure you want to change the base?

feat: Implement provider configuration system with full referenceability #53

Uh oh!

Conversation

eveld commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Provider Configuration with DAG Integration

Summary

Key Changes

Architectural Transformation

Provider Lifecycle Management

DAG Integration

Parser Improvements

Testing Improvements

Key Features Enabled

Advanced Use Cases

Integration

Test Plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eveld commented Jul 14, 2025 •

edited

Loading