Update wasi-testsuite to support testing newer versions of WASI

[wingo]

Hello wasi-testsuite hackers, I would like to update wasi-testsuite to support newer versions of WASI. The idea would be that the scope of wasi-testsuite should be to test the versions of WASI that are in use and in development, which I guess is currently wasip1, wasip2, and wasip3.

I am coordinating this work on #wasi > wasi-testsuite update for wasip3.

My plan first is to update the wasi-sdk used to generate the C test binaries, passing --target=wasm32-wasip1. There are some tests whose assertions should be relaxed; for example, testing with current wasi-sdk, I get ENOENT when opening a non-existent file instead of ENOTCAPABLE, and so I will change the test to relax the assertion on the precise error code. Then I will arrange for the C tests to compile with --target=wasm32-wasip2; the tests themselves shouldn't need to change. There will be some specific wasip2 tests, and specific tests for the in-progress wasip3 (which still uses the wasm32-wasip2 target triple).

This will mean that the set of directories to search for tests will change, e.g. instead of tests/c/testsuite, we might have tests/c/testsuite/wasm32-wasip1 and so on. I'll update the github workflows appropriately.

Then once the machinery is done to keep C wasip1 tests working, I'll do the same for rust, then start writing wasip3 tests, which will probably mostly be written in Rust because its toolchain is the most advanced for this use-case. The test runner itself will need to learn some wasip3-specific things, like how to compose components and so on. It's going to be a ✨ journey ✨ but I think we can get there and still keep everyone satisfied.

Thoughts welcome!

[pchickey]

Thanks @wingo for taking on this work. For some transparency: the BA has contracted Igalia to work on testing for wasip2 and wasip3. I am happy to serve as the reviewer for all new testing of wasip2 and wasip3, but I'll also ask @loganek to please check this work to make sure it doesn't cause any issues for this testsuite's wasip1 consumers.

[loganek]

Hi @wingo , thanks a lot for that. I'm aligned with expanding the purpose of this repo to cover other than p1 WASI versions. I agree that it'd require some restructuring; before jumping into code I'd suggest to come up with a plan/design that we can review and iterate on, and once we're all aligned on that, we can move to the implementation. Let me know if you're happy with that approach.

[wingo]

Greets @loganek! Up-front alignment sounds good to me :)

My plan as it stands is more of a sketch; I am still finding out more about the domain as I build things. I wonder if it might be sufficient to sync up on general principles. Let me share where I'm at, and perhaps you can let me know what you think or if something important is missing.

The overriding goal is to test wasip3 implementations. If we manage to do this here in wasi-testsuite, then we should do so while also keeping tests of wasip1 implementations working. There are two main questions to resolve: how to extend wasi-testsuite to support multiple WASI versions; and how to test waspi3, which adds asynchrony.

Some observations:

1. Sometimes between wasip1 and wasip3, the test-case code can remain the same. This is the case for, like, the C tests coded to POSIX APIs. I don't want to duplicate these tests in the repo.
2. For programs written directly against WASI interfaces, waspi1 and waspi3 tests are different. Pretty much all of the Rust cases fall into this category.
3. For wasip3, I am not sure if we will need to run tests involving more than a single component (besides what is provided by the host). If we can avoid it, that would simplify things.
4. To write wasip3 tests, my current plan is to mostly write the components in Rust and use wit-bindgen. Initial tests will be against wasmtime but I will also add jco/node once the test suite fills out a bit.
5. We can keep the "write the binaries to prod/testsuite-base" approach, and keep the nightly workflows working; it would be nice to surface those results somewhere tho!

As far as project structure and organization, I think we don't need much of a change: each language still has its build script, and instead of writing binaries to e.g. tests/c/testsuite, we can have tests/c/testsuite/wasm32-wasip1, etc as in #112. In the specific case of C, POSIX tests can remain the same and be separately compiled per-target. In the general case we may need e.g. tests/rust/src/wasip1 and tests/rust/src/wasip3 or so.

On the implementation side, we can keep the adapter python files, and add a command-line argument to select the WASI version and any optional features. I think this sort of detail is more amenable to review in PRs or chat and doesn't need to be part of an up-front design; what do you think?

Also, eventually it might make sense to add test-suite aliases at the top-level, e.g. to test all wasip3 things without having to separately pass tests/c/testsuite/wasm32-wasip3, tests/rust/testsuite/wasm32-wasip3, etc.

[wingo]

So @loganek if you are on board, the first part of this is making room for wasip3 tests. To that end I would suggest the following order of merges:

1. upload_test_binaries clears testsuite before downloading artifact #116. Otherwise if a non-wasm file is removed from e.g. tests/c/testsuite, it would stick around in the prod/testsuite-base branch.
2. Build C test cases via Python script to target-specific dir #112. Change to have tests/c/testsuite entirely generated by the build.py, with per-target subdirs, e.g. tests/c/wasm32-wasip1. A followup will add wasm32-wasip2.
3. Per-target rust testsuite #114. This does the same for Rust as we did for C, making tests/rust/testsuite completely generated, and putting the existing tests into tests/rust/wasm32-wasip1.
4. Per-target Assemblyscript testsuite #115. Do the same for Assemblyscript as for Rust and C.

With that, we will have the space to compile the POSIX C programs to wasip2, and to start adding wasip3 tests built using Rust.

[loganek]

Thanks, I'm aligned with the plan as discussed in Zulip.