Merge pull request #18 from miaow2/develop

miaow2 · web-flow · commit f5acb735fea3 · 2023-07-30T20:18:06.000+08:00
Added missing and extra config lines
diff --git a/.github/workflows/mkdocs.yaml b/.github/workflows/mkdocs.yaml
@@ -16,5 +16,7 @@ jobs:
       - uses: actions/setup-python@v4
         with:
           python-version: "3.10"
-      - run: python -m pip install -r requirements/docs.txt
+      - run: |
+            python -m pip install --upgrade pip
+            python -m pip install -r requirements/docs.txt
       - run: mkdocs gh-deploy --force
diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml
@@ -15,7 +15,7 @@ jobs:
       - name: Setup publish env
         run: |
             python -m pip install --upgrade pip
-            python -m pip install setuptools wheel build twine
+            python -m pip install -r requirements/publish.txt
       - name: Build
         run: python -m build
       - name: Check
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,17 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.3.0
+    hooks:
+      - id: check-yaml
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+  - repo: https://github.com/psf/black
+    rev: 23.1.0
+    hooks:
+      - id: black
+        args: [--check, --diff, --color, .]
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.0.280
+    hooks:
+      - id: ruff
+        args: [netbox_config_diff]
diff --git a/README.md b/README.md
@@ -1,3 +1,11 @@
+[![NetBox version](https://img.shields.io/badge/NetBox-3.5-blue.svg)](https://github.com/netbox-community/netbox)
+[![Supported Versions](https://img.shields.io/pypi/pyversions/netbox-config-diff.svg)](https://pypi.org/project/netbox-config-diff/)
+[![PyPI version](https://badge.fury.io/py/netbox-config-diff.svg)](https://badge.fury.io/py/netbox-config-diff)
+[![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![CI](https://github.com/miaow2/netbox-config-diff/actions/workflows/commit.yaml/badge.svg?branch=develop)](https://github.com/miaow2/netbox-config-diff/actions)
+
 # NetBox Config Diff Plugin
 
 NetBox plugin for Config Diff.
@@ -14,7 +22,7 @@ Read about [DataSources](https://demo.netbox.dev/static/docs/models/core/datasou
 Device configuration renders natively in NetBox. This [feature](https://demo.netbox.dev/static/docs/features/configuration-rendering/) was introduced in 3.5 version.
  NetBox Labs [blog](https://demo.netbox.dev/static/docs/features/configuration-rendering/) post about it.
 
-Plugin supports a wide list of vendors (Cisco, Juniper, Huawei, MicroTik etc.) with the help of Scrapli. Read [Scrapli](https://github.com/carlmontanari/scrapli/tree/main/scrapli/driver/core) and [scrapli-community](https://github.com/scrapli/scrapli_community/tree/main/scrapli_community) documentations to find full list of vendors.
+Plugin supports a wide list of vendors (Cisco, Juniper, Huawei, MicroTik etc.) with the help of Scrapli. Read [Scrapli](https://carlmontanari.github.io/scrapli/user_guide/project_details/#supported-platforms) and [scrapli-community](https://scrapli.github.io/scrapli_community/user_guide/project_details/#supported-platforms) documentations to find full list of vendors.
 <!--about-end-->
 
 ## Compatibility
diff --git a/docs/media/screenshots/compliance-missing-extra.png b/docs/media/screenshots/compliance-missing-extra.png
diff --git a/docs/usage.md b/docs/usage.md
@@ -5,7 +5,9 @@ Under `Plugins` navbar menu you can find plugin
 
 ![Screenshot of navbar](media/screenshots/navbar.png)
 
-Add PlatformSetting objects for your platforms in NetBox.
+## PlatformSetting
+
+Add PlatformSetting objects for your platforms in NetBox. This model contains info about how to connect and what to collect from device.
 
 Define:
 
@@ -19,6 +21,8 @@ You can test regexp on the site https://regex101.com/.
 
 ![Screenshot of PlatformSetting](media/screenshots/platformsetting.png)
 
+## Script
+
 Plugin adds a custom script `ConfigDiffScript` that runs all logic about diff calculations and connections to devices.
 You can find scripts list in navbar `Customization -> Scripts`.
 
@@ -43,12 +47,33 @@ If you have configs in NetBox DataSource, you can define it, the script instead
 
 ![Screenshot of the script](media/screenshots/script.png)
 
+## Results
+
 After script is done you can find results in `Config Compliances` menu. Each device has its own result.
 
 ![Screenshot of the compliance list](media/screenshots/compliance-list.png)
 
 Also result is storing rendered and actual configurations from devices.
 
+With the help of [netutils](https://github.com/networktocode/netutils) library plugin stores missing and extra config lines.
+
+![Screenshot of the missing/extra lines](media/screenshots/compliance-missing-extra.png)
+
+Supported platforms for missing/extra lines:
+
+* Arista EOS (arista_eos)
+* Aruba AOSCX (aruba_aoscx)
+* Cisco AireOS (cisco_aireos)
+* Cisco ASA (cisco_asa)
+* Cisco IOS-XE (cisco_iosxe)
+* Cisco IOS-XR (cisco_iosxr)
+* Cisco NX-OS (cisco_nxos)
+* Juniper JunOS (juniper_junos)
+* Mikrotik RouterOS (mikrotik_routeros)
+* Nokia SROS (nokia_sros)
+* PaloAlto PanOS (paloalto_panos)
+* Ruckus FastIron (ruckus_fastiron)
+
 Compliance finished with error
 
 ![Screenshot of the compliance error](media/screenshots/compliance-error.png)
@@ -59,4 +84,4 @@ Render diff between configurations
 
 No diff
 
-![Screenshot of the compliance ok](media/screenshots/compliance-ok.png)
+![Screenshot of the compliance ok](media/screenshots/compliance-ok.png)
diff --git a/netbox_config_diff/__init__.py b/netbox_config_diff/__init__.py
@@ -10,7 +10,7 @@ class ConfigDiffConfig(PluginConfig):
     verbose_name = "NetBox Config Diff Plugin"
     description = "Find diff between the intended device configuration and actual."
     author = __author__
-    email = __email__
+    author_email = __email__
     version = __version__
     base_url = "config-diff"
     required_settings = ["USERNAME", "PASSWORD"]
diff --git a/netbox_config_diff/api/serializers.py b/netbox_config_diff/api/serializers.py
@@ -24,6 +24,8 @@ class Meta:
             "diff",
             "rendered_config",
             "actual_config",
+            "missing",
+            "extra",
             "created",
             "last_updated",
         )
diff --git a/netbox_config_diff/compliance/base.py b/netbox_config_diff/compliance/base.py
@@ -9,15 +9,15 @@
 from django.core.exceptions import ObjectDoesNotExist
 from django.db.models import Q
 from extras.plugins import get_plugin_config
-from extras.querysets import ConfigContextQuerySet
 from extras.scripts import MultiObjectVar, ObjectVar
 from jinja2.exceptions import TemplateError
+from netutils.config.compliance import diff_network_config
 from utilities.exceptions import AbortScript
 
 from netbox_config_diff.models import ConfigCompliance
 
 from .models import DeviceDataClass
-from .utils import exclude_lines, get_unified_diff
+from .utils import PLATFORM_MAPPING, exclude_lines, get_unified_diff
 
 
 class ConfigDiffBase:
@@ -52,7 +52,7 @@ def run_script(self, data: dict) -> None:
         self.get_diff(devices)
         self.update_in_db(devices)
 
-    def validate_data(self, data: dict) -> Iterable[ConfigContextQuerySet]:
+    def validate_data(self, data: dict) -> Iterable[Device]:
         if not data["site"] and not data["devices"]:
             raise AbortScript("Define site or devices")
         if data.get("data_source") and data["data_source"].status != DataSourceStatusChoices.COMPLETED:
@@ -101,7 +101,7 @@ def log_results(self, device: DeviceDataClass) -> None:
         else:
             self.log_success(f"{device.name} no diff")
 
-    def get_devices_with_rendered_configs(self, devices: Iterable[ConfigContextQuerySet]) -> Iterator[DeviceDataClass]:
+    def get_devices_with_rendered_configs(self, devices: Iterable[Device]) -> Iterator[DeviceDataClass]:
         username = get_plugin_config("netbox_config_diff", "USERNAME")
         password = get_plugin_config("netbox_config_diff", "PASSWORD")
         for device in devices:
@@ -151,6 +151,12 @@ def get_diff(self, devices: list[DeviceDataClass]) -> None:
         for device in devices:
             if device.error is not None:
                 continue
-            device.diff = get_unified_diff(
-                device.rendered_config, exclude_lines(device.actual_config, device.exclude_regex), device.name
-            )
+            cleaned_config = exclude_lines(device.actual_config, device.exclude_regex)
+            device.diff = get_unified_diff(device.rendered_config, cleaned_config, device.name)
+            if device.platform in PLATFORM_MAPPING:
+                device.missing = diff_network_config(
+                    device.rendered_config, cleaned_config, PLATFORM_MAPPING[device.platform]
+                )
+                device.extra = diff_network_config(
+                    cleaned_config, device.rendered_config, PLATFORM_MAPPING[device.platform]
+                )
diff --git a/netbox_config_diff/compliance/models.py b/netbox_config_diff/compliance/models.py
@@ -19,6 +19,8 @@ class DeviceDataClass:
     rendered_config: str | None = None
     actual_config: str | None = None
     diff: str | None = None
+    missing: str | None = None
+    extra: str | None = None
     error: str | None = None
     auth_strict_key: bool = False
     transport: str = "asyncssh"
@@ -51,6 +53,8 @@ def to_db(self):
             "error": self.error or "",
             "rendered_config": self.rendered_config or "",
             "actual_config": self.actual_config or "",
+            "missing": self.missing or "",
+            "extra": self.extra or "",
         }
 
     async def get_actual_config(self):
diff --git a/netbox_config_diff/compliance/utils.py b/netbox_config_diff/compliance/utils.py
@@ -1,10 +1,29 @@
 import re
 from difflib import unified_diff
 
+PLATFORM_MAPPING = {
+    "arista_eos": "arista_eos",
+    "cisco_aireos": "cisco_aireos",
+    "aruba_aoscx": "aruba_aoscx",
+    "cisco_asa": "cisco_asa",
+    "cisco_iosxe": "cisco_ios",
+    "cisco_iosxr": "cisco_iosxr",
+    "cisco_nxos": "cisco_nxos",
+    "juniper_junos": "juniper_junos",
+    "mikrotik_routeros": "mikrotik_routeros",
+    "nokia_sros": "nokia_sros",
+    "paloalto_panos": "paloalto_panos",
+    "ruckus_fastiron": "ruckus_fastiron",
+}
+
 
 def get_unified_diff(rendered_config: str, actual_config: str, device: str) -> str:
     diff = unified_diff(
-        rendered_config.splitlines(), actual_config.splitlines(), fromfiledate=device, tofiledate=device, lineterm=""
+        rendered_config.splitlines(),
+        actual_config.splitlines(),
+        fromfiledate=device,
+        tofiledate=device,
+        lineterm="",
     )
     return "\n".join(diff).strip()
 
diff --git a/netbox_config_diff/migrations/0005_configcompliance_extra_missing.py b/netbox_config_diff/migrations/0005_configcompliance_extra_missing.py
@@ -0,0 +1,20 @@
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ('netbox_config_diff', '0004_update_script'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='configcompliance',
+            name='extra',
+            field=models.TextField(blank=True),
+        ),
+        migrations.AddField(
+            model_name='configcompliance',
+            name='missing',
+            field=models.TextField(blank=True),
+        ),
+    ]
diff --git a/netbox_config_diff/models.py b/netbox_config_diff/models.py
@@ -31,6 +31,12 @@ class ConfigCompliance(ChangeLoggingMixin, models.Model):
     rendered_config = models.TextField(
         blank=True,
     )
+    missing = models.TextField(
+        blank=True,
+    )
+    extra = models.TextField(
+        blank=True,
+    )
 
     objects = RestrictedQuerySet.as_manager()
 
@@ -67,8 +73,8 @@ class PlatformSetting(NetBoxModel):
         max_length=25,
         help_text=_(
             "Scrapli driver for platfrom, you can find them in "
-            "<a href='https://github.com/carlmontanari/scrapli'>Scrapli</a> and "
-            "<a href='https://github.com/scrapli/scrapli_community'>Scrapli community</a> documentation."
+            "<a href='https://carlmontanari.github.io/scrapli/user_guide/project_details/#supported-platforms'>Scrapli</a> and "  # noqa
+            "<a href='https://scrapli.github.io/scrapli_community/user_guide/project_details/#supported-platforms'>Scrapli community</a> documentation."  # noqa
         ),
     )
     command = models.CharField(
diff --git a/netbox_config_diff/templates/netbox_config_diff/configcompliance/missing_extra.html b/netbox_config_diff/templates/netbox_config_diff/configcompliance/missing_extra.html
@@ -0,0 +1,42 @@
+{% extends "netbox_config_diff/configcompliance/base.html" %}
+
+{% block title %}{{ object }} - Missing/Extra{% endblock %}
+
+{% block content %}
+  <div class="row">
+    <div class="col col-md-6">
+      <div class="card">
+        <div class="card-header">
+          <div class="float-end">
+            <a href="?export_missing=True" class="btn btn-sm btn-primary" role="button">
+              <i class="mdi mdi-download" aria-hidden="true"></i> Download
+            </a>
+          </div>
+          <h5>Missing</h5>
+        </div>
+        {% if object.missing %}
+          <pre class="card-body">{{ object.missing }}</pre>
+        {% else %}
+          <div class="card-body text-muted">No lines</div>
+        {% endif %}
+      </div>
+    </div>
+    <div class="col col-md-6">
+      <div class="card">
+        <div class="card-header">
+          <div class="float-end">
+            <a href="?export_extra=True" class="btn btn-sm btn-primary" role="button">
+              <i class="mdi mdi-download" aria-hidden="true"></i> Download
+            </a>
+          </div>
+          <h5>Extra</h5>
+        </div>
+        {% if object.extra %}
+          <pre class="card-body">{{ object.extra }}</pre>
+        {% else %}
+          <div class="card-body text-muted">No lines</div>
+        {% endif %}
+      </div>
+    </div>
+  </div>
+{% endblock %}
diff --git a/netbox_config_diff/views.py b/netbox_config_diff/views.py
@@ -75,6 +75,42 @@ class ConfigComplianceActualConfigView(BaseConfigComplianceConfigView):
     )
 
 
+@register_model_view(ConfigCompliance, "missing-extra")
+class ConfigComplianceMissingExtraConfigView(generic.ObjectView):
+    queryset = ConfigCompliance.objects.all()
+    template_name = "netbox_config_diff/configcompliance/missing_extra.html"
+    tab = ViewTab(
+        label=_("Missing/Extra"),
+        weight=520,
+    )
+
+    def export_parts(self, name, lines, suffix):
+        response = HttpResponse(lines, content_type="text")
+        filename = f"{name}_{suffix}.txt"
+        response["Content-Disposition"] = f'attachment; filename="{filename}"'
+        return response
+
+    def get(self, request, **kwargs):
+        instance = self.get_object(**kwargs)
+        context = self.get_extra_context(request, instance)
+
+        if request.GET.get("export_missing"):
+            return self.export_parts(instance.device.name, instance.missing, "missing")
+
+        if request.GET.get("export_extra"):
+            return self.export_parts(instance.device.name, instance.extra, "extra")
+
+        return render(
+            request,
+            self.get_template_name(),
+            {
+                "object": instance,
+                "tab": self.tab,
+                **context,
+            },
+        )
+
+
 class ConfigComplianceListView(generic.ObjectListView):
     queryset = ConfigCompliance.objects.prefetch_related("device")
     filterset = ConfigComplianceFilterSet
diff --git a/requirements/base.txt b/requirements/base.txt
@@ -1,2 +1,3 @@
+netutils==1.5.0
 scrapli[asyncssh]==2023.01.30
-scrapli-community==2023.01.30
+scrapli-community==2023.01.30
diff --git a/requirements/publish.txt b/requirements/publish.txt
@@ -0,0 +1,2 @@
+build==0.10.0
+twine==4.0.2

Original file line number	Diff line number	Diff line change
`@@ -24,6 +24,8 @@ class Meta:`
`24`	`24`	`"diff",`
`25`	`25`	`"rendered_config",`
`26`	`26`	`"actual_config",`
	`27`	`+ "missing",`
	`28`	`+ "extra",`
`27`	`29`	`"created",`
`28`	`30`	`"last_updated",`
`29`	`31`	`)`