SpaceAPI Validator v3 API

[s3lph]

In #70 I proposed an amendment to the validator API v2 to add support for separate IPv6 and IPv4 checks as well as HTTP status codes. After some initial feedback and further thought I realized that a new major version of the validatorAPI might be the better approach. Therefore, I had closed #70 and started work on drafting a specification for a v3 API.

Even though I have already added an (so far still incomplete) implementation of the proposed v3 API, since it didn't require too much change from v2 in the server code, I would like to first focus discussion on the API specification itself.

The major changes between v2 and my proposal for v3 are:

• The ValidateJsonV3Response component makes the message key optional (see v2: Message field in response #50). However, the checkedVersions key becomes required (though it may be empty in case the validated JSON file did not actually declare compliance with any supported schema version).
• A single call to /v3/validateURL can perform multiple validation runs using multiple parameters. The parameters currently included in my draft are:
  • IP address family (IPv6, IPv4, or any): "any" describes the same behavior as the current v2 implementation, which just leaves protocol selection up to the network stack. However, doing validation once via IPv6 and once via IPv4 allows us to discover hard to reproduce endpoint issues related to one address family only.
  • HTTP User Agent: Testing with a few different user agent strings (especially commonly blocked ones) allows us to troubleshoot issues related to user agent blocking such as directory#288 a lot easier.
• The ValidateUrlV3 component adds two new keys:
  • extendedValidation: If true, instruct the validator to perform "extended validation" (such as IP dual stack testing, or testing with different user agent strings). The exact set of additional validation scenarios is implementation dependent.
  • withValidatedJson: If false, instruct the validator not to return the JSON body of the SpaceAPI endpoint that is being validated.
• The biggest change was done to the ValidateUrlV3Response component: Most fields were removed from the component itself, and moved to the newly introduced ValidateUrlV3Result component, which the response contains a list of. The ValidateUrlV3Result component is effectively what ValidateUrlV2Response currently is in V2. Each ValidateUrlV3Result describes the validation result under a certain set of parameters (at the moment: IP address family and user agent). The parameters themselves are contained in the result as well as ValidateUrlV3Parameters.
• To reduce duplication and discrepancies between JSON validation and URL validation, the actual SpaceAPI schema-relevant validation fields (valid, checkedVersions, message, schemaErrors) are removed from ValidateUrlV3Result, and are moved to an instance of ValidateJsonV3Response embedded in ValidateUrlV3Result.
• ValidateUrlV3Result also adds some new fields not present in ValidateUrlV2Response:
  • The HTTP status code that was served by the endpoint.
  • The original and optionally redirected URL
  • Whether redirects between the submitted URL and the final URL have all been permanent redirects.

This PR also includes the addition of a swagger UI for the validator API.

I'm aware that this adds quite a bit of complexity to the validator API, however, I think this allows us to both improve validation and use the validator results for a few more use cases:

• This allows us to easily troubleshoot hard-to-spot endpoint issues with regard to network protocol or user agent.
• The collection of HTTP status code and redirection allows us to easily spot issues such as an URL being submitted to the directory that already redirects to something else.
• Since we validate all URLs in the directory on a regular basis and collect the validation result in a TSDB, this extension of the validator could be used to automate directory housekeeping tasks, e.g. "remove endpoints that have served a 410 Gone consistently for $time", or "update endpoint URLS that have served permanent redirects consistently for $time".

I'd be happy to get some feedback on my proposal.

/v3/validateURL extended validation usage example

curl -XPOST http://127.0.0.1:8080/v3/validateURL -H 'Content-Type: application/json' -d '{"url":"http://spaceapi.kabelsalat.ch","extendedValidation":true,"withValidatedJson":false}' | jq
{
  "results": [
    {
      "parameters": {
        "addressFamily": "ipv6",
        "userAgent": "SpaceApi Validator/1.3.0 (https://github.com/SpaceApi/validator)"
      },
      "originalUrl": "http://spaceapi.kabelsalat.ch",
      "redirectedUrl": "https://spaceapi.kabelsalat.ch/",
      "reachable": true,
      "httpStatus": 200,
      "isHttps": false,
      "httpsRedirect": true,
      "permanentRedirect": true,
      "cors": true,
      "contentType": true,
      "invalidCert": false,
      "validation": {
        "valid": true,
        "checkedVersions": [
          "14",
          "15",
          "16"
        ]
      }
    },
    {
      "parameters": {
        "addressFamily": "ipv4",
        "userAgent": "SpaceApi Validator/1.3.0 (https://github.com/SpaceApi/validator)"
      },
      "originalUrl": "http://spaceapi.kabelsalat.ch",
      "redirectedUrl": "https://spaceapi.kabelsalat.ch/",
      "reachable": true,
      "httpStatus": 200,
      "isHttps": false,
      "httpsRedirect": true,
      "permanentRedirect": true,
      "cors": true,
      "contentType": true,
      "invalidCert": false,
      "validation": {
        "valid": true,
        "checkedVersions": [
          "14",
          "15",
          "16"
        ]
      }
    },
    {
      "parameters": {
        "addressFamily": "any",
        "userAgent": "curl/8.16.0"
      },
      "originalUrl": "http://spaceapi.kabelsalat.ch",
      "redirectedUrl": "https://spaceapi.kabelsalat.ch/",
      "reachable": true,
      "httpStatus": 200,
      "isHttps": false,
      "httpsRedirect": true,
      "permanentRedirect": true,
      "cors": true,
      "contentType": true,
      "invalidCert": false,
      "validation": {
        "valid": true,
        "checkedVersions": [
          "14",
          "15",
          "16"
        ]
      }
    },
    {
      "parameters": {
        "addressFamily": "any",
        "userAgent": "Go-http-client/1.1"
      },
      "originalUrl": "http://spaceapi.kabelsalat.ch",
      "redirectedUrl": "https://spaceapi.kabelsalat.ch/",
      "reachable": true,
      "httpStatus": 200,
      "isHttps": false,
      "httpsRedirect": true,
      "permanentRedirect": true,
      "cors": true,
      "contentType": true,
      "invalidCert": false,
      "validation": {
        "valid": true,
        "checkedVersions": [
          "14",
          "15",
          "16"
        ]
      }
    }
  ],
  "reachable": true,
  "valid": true
}