libfetch, fetch: Improve handling of HTTP request body, method, fields, and status code

[ryan-moeller]

Modern APIs often rely on HTTP fields as part of their interface, both
in requests and responses.

lib/libfetch: Add fetchXReqHTTP(3)

Add a function to libfetch allowing additional request fields to be
specified for an HTTP request and response fields to be returned to the
caller.

This function accepts an optional stream for the body content, as a
nul-terminated C string may be inadequate for some content. If the
length of the stream can be determined, a Content-Length header is
added. Otherwise, the body is sent chunk-encoded.

usr.bin/fetch: Handle custom headers and method for HTTP

Add functionality to fetch allowing HTTP method and headers to be
specified. The option names align with those used by curl.

usr.bin/fetch: Add options for HTTP upload data

Add a -D/--data option to specify a file from which to read HTTP request
body contents, with the special case that "-" will read from stdin.

This option deviates from curl, taking a much more barebones approach
for simplicity.

Add a -C/--chunked option to specify that the HTTP request body contents
are to use chunked transfer-encoding. Otherwise, it is automatically
used by libfetch when the size of the file cannot be predetermined.

lib/libfetch: Accept all HTTP 2xx (Successful) status codes

Some HTTP 2xx (Successful) class status codes were being treated as
errors. Add a definition for the full range of successful status codes
and use it to test for a success response.

This allows libfetch and fetch to receive the response body from more
practical requests, such as POST responding 201 Created with content.

Demonstration

Here is an example of the new functionality being used to access the GitHub REST API. The API requires a valid bearer token in the Authorization header, along with a couple other recommended headers described in the documentation.

$ fetch -q \
    -X GET \
    -H "Accept: application/vnd.github+json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    -o - \
    https://api.github.com/repos/ryan-moeller/kernel-nbd-client/branches
[{"name":"main","commit":{"sha":"d69b0a1c8d2b63ab48726027e6e756ccf9a0f2fb","url":"https://api.github.com/repos/ryan-moeller/kernel-nbd-client/commits/d69b0a1c8d2b63ab48726027e6e756ccf9a0f2fb"},"protected":false},{"name":"structured","commit":{"sha":"4831ee5ece393e596262d76609a0be69d3d408bf","url":"https://api.github.com/repos/ryan-moeller/kernel-nbd-client/commits/4831ee5ece393e596262d76609a0be69d3d408bf"},"protected":false},{"name":"unix/stream","commit":{"sha":"7f020b22ec0e9414a212fe34b01701b23443d825","url":"https://api.github.com/repos/ryan-moeller/kernel-nbd-client/commits/7f020b22ec0e9414a212fe34b01701b23443d825"},"protected":false}]

The (redundant, as it is the default) GET method is specified along with a valid token in an Authorization header, and the server responds with the requested data.

Specifying a different HTTP method is allowed by fetch(1) and rejected by the server as expected:

$ fetch -q \
    -X TEST \
    -H "Accept: application/vnd.github+json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    -o - \
    https://api.github.com/repos/ryan-moeller/kernel-nbd-client/branches
fetch: https://api.github.com/repos/ryan-moeller/kernel-nbd-client/branches: Forbidden

The server does not respond to requests with an invalid token:

$ fetch -q \
    -H "Accept: application/vnd.github+json" \
    -H "Authorization: Bearer not$GITHUB_TOKEN" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    -o - \
    https://api.github.com/repos/ryan-moeller/kernel-nbd-client/branches
fetch: https://api.github.com/repos/ryan-moeller/kernel-nbd-client/branches:
$ echo $?
1

This is not the most compelling example, as this is a read-only use of the GitHub API to access public data that would have been allowed with no token. There are of course other API endpoints with stricter access controls where the headers are actually necessary. And this functionality becomes even more useful with support for fetch(1) uploading a request body. This enables use of common HTTP methods like PUT, POST, and PATCH.

For example, uploading a gist to GitHub uses most of the new features:

$ fetch -q \
    -X POST \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -H "Accept: application/vnd.github+json" \
    -H "Content-Type: application/json" \
    -D - \
    -o - \
    https://api.github.com/gists <<EOF
{
  "description": "fetch(1) test gist",
  "public": false,
  "files": {
    "demo.txt": {
      "content": "This gist was created using fetch(1)"
    }
  }
}
EOF
<verbose JSON response omitted>

-X POST tells fetch to use the POST method for the HTTP request
-H ... adds custom headers to the request
-D - sends a request body from a file (stdin, in this case)

Testing

FreeBSD has no tests for libfetch or fetch. I didn't go so far as to implement a Kyua test suite for inclusion in FreeBSD, but I do have some ancillary testing apparatus used to exercise my changes.

My test harness for this uses a Lua library wrapping libfetch with a Lua script exercising all request and response field sections as well as different request and response body transfer encodings and different success status codes. This can all be found here.

CI runs: https://cirrus-ci.com/build/5568610090876928
15.0-ALPHA4 test output: https://api.cirrus-ci.com/v1/task/5673940506378240/logs/libfetch_test.log

[ryan-moeller]

The #ifdef _SYS_QUEUE_H_ is a compromise. Currently, fetch.h includes none of the headers it relies on. Rather than changing this, or changing every consumer to include the sys/queue.h header, I added checks for it to only expose the parts that rely on it when the consumer has included it.

I'm open to handling this differently.

[ryan-moeller]

This function is not used outside http.c so while changing the parameters I also made it static.

[ryan-moeller]

Seems worth replacing this warning with useful functionality. -H matches curl.

[ryan-moeller]

• Switched from using a struct iovec array for the body to support binary content, to using a FILE * and implementing automatic (or forced) chunked transfer-encoding. This enables streaming data uploads (in libfetch, not yet in fetch). Trailing headers remain unimplemented.

[ryan-moeller]

• Documented the 'C' flag in fetch.3

[ryan-moeller]

• Implemented body upload options for fetch(1)

[ryan-moeller]

• Fixed sorting of additions to fetch.3 so the convention of fetchXFoo being listed before fetchFoo is maintained

[ryan-moeller]

It would be best to implement support for HTTP trailer fields in libfetch while I'm here. Taking both headers and trailers into consideration, we would prefer to use more general terms like struct http_field, fields, struct http_fields, and fetchFreeResFields. Two additional optional parameters to fetchXReqHTTP would provide the request trailers and receive the response trailers.

I have started implementing this change and will update this pull request after I finish testing the idea.

[ryan-moeller]

• Revised names to more use generic "field" terms in libfetch
• Implemented request and response trailer fields in libfetch

I have a Lua binding for libfetch and a test server scripted with socat I've been testing with. I'll share more of the test harness details when I have some time.

[ryan-moeller]

I have a Lua binding for libfetch and a test server scripted with socat I've been testing with. I'll share more of the test harness details when I have some time.

• Brief description of testing apparatus added to the opening comment

[github-actions]

Thank you for taking the time to contribute to FreeBSD!
All issues resolved.

[ryan-moeller]

On closer inspection, https://www.rfc-editor.org/rfc/rfc9112#section-6.3-4.1 notes that request messages are never close-delimited. The previous commit has been removed.

[ryan-moeller]

I've just noticed that the group of HTTP response codes to treat as success in libfetch is missing a few. I'll be back with another revision to treat e.g. 201 Created as success and proceed to read the rest of the response. That will inspire some new test cases, as well.

[ryan-moeller]

• Allow libfetch and fetch to receive the response body for more status codes, such as 201 Created

[ryan-moeller]

• nit: In the first commit, make fetchReqHTTP use fetchXReqHTTP instead of http_request_body, like fetchGetHTTP uses fetchXGetHTTP instead of http_request.

[ryan-moeller]

• Rebased

[ryan-moeller]

I have another small commit extending the existing handling of Range and Content-Range headers in libfetch to better support range requests and partial responses according to RFC 9110 Section 14. However, I'm reluctant to continue expanding the original scope of this pull request. I'll hold off on pushing that commit unless requested, otherwise I can save it for another pull request.