[CHORE]: Implement automatic API documentation generation using mkdocstrings and update Makefile

[crivetimihai]

🧭 Chore Summary

Implement automatic API documentation generation using mkdocstrings: make docs-api to generate comprehensive API docs from docstrings and integrate seamlessly with existing MkDocs site while maintaining 100% coverage of public APIs.

---

🧱 Areas Affected

• Documentation build system / Make targets (make docs-api, make docs-serve, make docs-deploy)
• MkDocs configuration (mkdocs.yml, plugin setup)
• GitHub Actions / CI pipeline (docs deployment)
• Documentation structure (docs/docs/api/ organization)
• Python package structure (docstring quality, __all__ exports)
• Dependencies (requirements.txt for mkdocstrings)

---

⚙️ Context / Rationale

Automatic API documentation ensures developers always have current, accurate reference material. mkdocstrings reads Python docstrings and generates beautiful, searchable API docs that integrate seamlessly with your existing MkDocs site. This eliminates manual API documentation maintenance and provides comprehensive coverage of all classes, functions, and modules.

What is mkdocstrings?
mkdocstrings is a MkDocs plugin that automatically generates API documentation from your Python source code. It parses docstrings, type hints, and source code to create comprehensive reference documentation.

Basic Integration Example:

# mkdocs.yml
plugins:
  - mkdocstrings:
      handlers:
        python:
          options:
            show_source: true
            show_root_heading: true

Simple API Reference Page:

# API Reference

## Core Gateway Classes

::: mcpgateway.gateway.Gateway
    options:
      show_source: true
      show_root_heading: true
      
::: mcpgateway.connection.MCPConnection
    options:
      show_source: false
      members_order: source

Advanced Configuration with Filters:

plugins:
  - mkdocstrings:
      handlers:
        python:
          paths: [mcpgateway]
          options:
            show_source: true
            show_root_heading: true
            show_root_toc_entry: false
            show_category_heading: true
            group_by_category: true
            filters:
              - "!^_"  # Exclude private members
              - "!^__(?!init)__"  # Exclude magic methods except __init__
            docstring_style: google
            docstring_options:
              ignore_init_summary: true
            merge_init_into_class: true

MCPGateway Specific Examples:

# Gateway API Reference

## Core Components

### Gateway Management
::: mcpgateway.gateway.Gateway
    options:
      show_source: true
      members_order: source
      filters: ["!^_"]

### Connection Handling  
::: mcpgateway.connection.MCPConnection
    options:
      show_source: false
      group_by_category: true

### Request Routing
::: mcpgateway.routing.RequestRouter
    options:
      show_root_heading: true
      show_category_heading: true

## API Schemas

### Tool Schemas
::: mcpgateway.schemas.ToolCreate
::: mcpgateway.schemas.ToolRead
::: mcpgateway.schemas.ToolUpdate

### Gateway Schemas  
::: mcpgateway.schemas.GatewayCreate
::: mcpgateway.schemas.GatewayRead

### Server Management
::: mcpgateway.schemas.ServerCreate
::: mcpgateway.schemas.ServerRead

## Services

### Tool Service
::: mcpgateway.services.tool_service
    options:
      show_source: true
      filters: ["!^_"]

### Resource Service
::: mcpgateway.services.resource_service
    options:
      group_by_category: true

## Database Models

### Core Models
::: mcpgateway.models.tool.Tool
::: mcpgateway.models.gateway.Gateway
::: mcpgateway.models.server.Server

---

📦 Related Make Targets

Target	Purpose
make docs-api	Generate API documentation from docstrings
make docs-build	Build complete documentation site (existing + API docs)
make docs-serve	Serve documentation locally with live reload
make docs-deploy	Deploy documentation to GitHub Pages
make docs-check	Validate API documentation coverage and links
make docs-clean	Clean generated documentation files
make docs-api-coverage	Check API documentation coverage percentage
make docs-linkcheck	Validate all internal and external links
make docs-pdf	Generate PDF version of documentation (optional)

Bold targets are mandatory; CI must fail if API documentation is incomplete or build fails.

---

📋 Acceptance Criteria

• make docs-api generates comprehensive API documentation for all public classes/functions.
• make docs-build successfully builds complete documentation site including API reference.
• make docs-serve starts local development server with live reload functionality.
• make docs-check validates 100% API documentation coverage for public interfaces.
• API documentation integrates seamlessly with existing docs/docs/ structure.
• GitHub Actions builds and deploys documentation automatically on main branch pushes.
• All public APIs have proper docstrings with parameters, return values, and examples.
• Navigation includes clear API reference section organized by module/functionality.
• Search functionality works across both manual docs and auto-generated API docs.
• Changelog entry under "Documentation".

---

🛠️ Task List (suggested flow)

1. Baseline assessment

   # Check current docstring coverage
   find mcpgateway -name "*.py" -exec grep -l '"""' {} \; | wc -l
   find mcpgateway -name "*.py" | wc -l
   
   # Identify modules needing better docstrings
   python -c "
   import ast, os
   for root, dirs, files in os.walk('mcpgateway'):
       for file in files:
           if file.endswith('.py'):
               print(f'Checking {os.path.join(root, file)}')
   "

2. MkDocs configuration

   Update docs/mkdocs.yml:

   plugins:
     - search
     - mkdocstrings:
         default_handler: python
         handlers:
           python:
             paths: [.]
             options:
               show_source: true
               show_root_heading: true
               show_root_toc_entry: false
               show_category_heading: true
               group_by_category: true
               filters:
                 - "!^_"
                 - "!^__(?!init)__"
               docstring_style: google
               merge_init_into_class: true
   
   nav:
     - Home: index.md
     - API Reference:
       - Overview: api/index.md
       - Core: api/core.md
       - Services: api/services.md
       - Models: api/models.md
       - Schemas: api/schemas.md
     # ... existing navigation

3. Documentation structure

   Create API documentation organization:

   mkdir -p docs/docs/api
   
   # Create API overview page
   cat > docs/docs/api/index.md << 'EOF'
   # API Reference
   
   This section provides comprehensive API documentation for the MCPGateway.
   
   ## Quick Navigation
   
   - **[Core Components](core.md)** - Gateway, Connection, Routing
   - **[Services](services.md)** - Business logic and operations  
   - **[Models](models.md)** - Database models and relationships
   - **[Schemas](schemas.md)** - API request/response schemas
   
   ## Getting Started
   
   The MCPGateway API is organized into several key areas:
   
   1. **Core**: Essential classes for gateway functionality
   2. **Services**: High-level business logic and operations
   3. **Models**: Database models using SQLAlchemy
   4. **Schemas**: Pydantic models for API validation
   EOF

4. Makefile integration

   Add to existing docs/Makefile:

   .PHONY: docs-api docs-build docs-serve docs-deploy docs-check docs-clean
   
   docs-api:
   	@echo "📚 Generating API documentation..."
   	@cd .. && python -m mkdocs build --config-file docs/mkdocs.yml --site-dir docs/site
   
   docs-build: docs-api
   	@echo "🏗️  Building complete documentation site..."
   	@cd .. && python -m mkdocs build --config-file docs/mkdocs.yml --clean
   
   docs-serve:
   	@echo "🌐 Serving documentation locally..."
   	@cd .. && python -m mkdocs serve --config-file docs/mkdocs.yml --dev-addr localhost:8000
   
   docs-deploy:
   	@echo "🚀 Deploying documentation to GitHub Pages..."
   	@cd .. && python -m mkdocs gh-deploy --config-file docs/mkdocs.yml --force
   
   docs-check:
   	@echo "✅ Validating API documentation coverage..."
   	@python -c "
   	import ast, os, sys
   	total_classes = total_functions = documented = 0
   	for root, dirs, files in os.walk('../mcpgateway'):
   	    for file in files:
   	        if file.endswith('.py'):
   	            # Check docstring coverage logic here
   	            pass
   	coverage = (documented / (total_classes + total_functions)) * 100
   	print(f'API Documentation Coverage: {coverage:.1f}%')
   	if coverage < 90:
   	    sys.exit(1)
   	"
   
   docs-clean:
   	@echo "🧹 Cleaning generated documentation..."
   	@rm -rf site/ .cache/

5. Dependencies

   Update docs/requirements.txt:

   mkdocs-material>=9.0.0
   mkdocstrings[python]>=0.20.0
   mkdocs-gen-files>=0.4.0
   mkdocs-literate-nav>=0.6.0
   mkdocs-section-index>=0.3.0
   mkdocstrings-python>=0.8.0

6. API documentation pages

   Create comprehensive API reference:

   # Core components
   cat > docs/docs/api/core.md << 'EOF'
   # Core Components
   
   ## Gateway
   ::: mcpgateway.gateway.Gateway
       options:
         show_source: true
         filters: ["!^_"]
   
   ## Connection Management
   ::: mcpgateway.connection.MCPConnection
       options:
         group_by_category: true
   
   ## Request Routing  
   ::: mcpgateway.routing.RequestRouter
       options:
         show_category_heading: true
   EOF
   
   # Services
   cat > docs/docs/api/services.md << 'EOF'
   # Services
   
   ## Tool Service
   ::: mcpgateway.services.tool_service
       options:
         show_source: true
         filters: ["!^_"]
   
   ## Resource Service
   ::: mcpgateway.services.resource_service
   
   ## Server Service
   ::: mcpgateway.services.server_service
   
   ## Prompt Service
   ::: mcpgateway.services.prompt_service
   EOF

7. CI integration

   Add to existing GitHub Actions workflow:

   # Add to your existing lint matrix or create new job
   docs-build:
     name: 📚 Build Documentation
     runs-on: ubuntu-latest
     steps:
       - name: ⬇️  Checkout source
         uses: actions/checkout@v4
         with:
           fetch-depth: 0  # Full history for git info
   
       - name: 🐍  Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: "3.12"
           cache: pip
   
       - name: 📦  Install dependencies
         run: |
           python -m pip install --upgrade pip
           pip install -e .[dev]
           pip install -r docs/requirements.txt
   
       - name: 🔍  Check API documentation coverage
         run: make -C docs docs-check
   
       - name: 🏗️  Build documentation
         run: make -C docs docs-build
   
       - name: 🚀  Deploy to GitHub Pages
         if: github.ref == 'refs/heads/main'
         run: make -C docs docs-deploy
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

8. Docstring quality improvements

   Enhance existing docstrings for better API docs:

   class Gateway:
       """Central gateway for managing MCP server connections and routing.
       
       The Gateway class provides the main entry point for:
       - Managing multiple MCP server connections
       - Routing requests to appropriate servers  
       - Handling authentication and authorization
       - Providing unified API endpoints
       
       Attributes:
           connections: Dictionary of active MCP connections
           router: Request router for path-based routing
           auth_handler: Authentication/authorization handler
           
       Example:
           >>> gateway = Gateway()
           >>> gateway.add_server("tools", "ws://localhost:8080")
           >>> response = gateway.route_request("/tools/calculator", {...})
       """
   
   def route_request(self, path: str, payload: dict) -> dict:
       """Route a request to the appropriate MCP server.
       
       Args:
           path: Request path (e.g., "/tools/calculator")
           payload: Request payload to forward
           
       Returns:
           Response from the target MCP server
           
       Raises:
           ValueError: If no route matches the path
           ConnectionError: If target server is unavailable
           
       Example:
           >>> response = gateway.route_request("/api/v1/tools", {
           ...     "method": "tools/call",
           ...     "params": {"name": "calculator", "args": {"x": 5, "y": 3}}
           ... })
           >>> response["result"]
           8
       """

9. Advanced mkdocstrings features

   Configure advanced documentation features:

   # In mkdocs.yml
   plugins:
     - mkdocstrings:
         handlers:
           python:
             options:
               # Cross-references
               show_signature_annotations: true
               signature_crossrefs: true
               
               # Source code
               show_source: true
               show_bases: true
               
               # Headings and structure  
               heading_level: 2
               show_root_heading: true
               show_root_members_full_path: false
               
               # Filtering
               inherited_members: true
               filters:
                 - "!^_"
                 - "!^__(?!init)__" 
               
               # Docstring processing
               docstring_style: google
               docstring_options:
                 ignore_init_summary: true
               
               # Rendering
               separate_signature: true
               merge_init_into_class: true
               show_submodules: true

10. Final validation

    make -C docs docs-clean
    make -C docs docs-check
    make -C docs docs-build
    make -C docs docs-serve  # Test locally

---

📖 References

• mkdocstrings – Automatic documentation from sources · https://mkdocstrings.github.io/
• mkdocstrings-python – Python handler for mkdocstrings · https://mkdocstrings.github.io/python/
• MkDocs Material – Material Design theme for MkDocs · https://squidfunk.github.io/mkdocs-material/
• MkDocs – Fast, simple and clean documentation generator · https://www.mkdocs.org/
• Google Docstring Style – Documentation conventions · https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings

---

🧩 Additional Notes

• Integration with existing docs: API docs will be seamlessly integrated into your current docs/docs/ structure without disrupting existing content.
• Incremental deployment: Start with core modules and gradually expand API documentation coverage.
• Live reload: make docs-serve provides instant preview of documentation changes during development.
• Cross-references: mkdocstrings automatically creates links between related classes and functions.
• Type hint integration: Leverages your existing type hints for enhanced API documentation.
• Search optimization: Generated API docs are fully searchable through MkDocs' built-in search.
• Mobile responsive: API documentation works perfectly on all devices thanks to Material Design theme.

Documentation Best Practices:

• Use Google-style docstrings for consistency with mkdocstrings defaults
• Include comprehensive examples in docstrings, especially for complex APIs
• Organize API reference by functional areas (core, services, models, schemas)
• Keep manual documentation in sync with auto-generated API docs
• Use __all__ exports to control which items appear in API documentation
• Leverage type hints - they automatically appear in generated documentation