Implement IDL tests

[kainino0x]

IDL tests can be highly automated. WPT appears to have a helper for this: idlharness.js.

We should see about using that instead of writing any of our own IDL tests.

See also previous notes: https://hackmd.io/@webgpu/SJtUks9CU

[kainino0x]

Note, these tests are not a high priority, because they only verify rules that should happen automatically in any compliant WebIDL generator.

[kainino0x]

It may be the case that there is infrastructure already out there somewhere to generate these tests.

gpuweb/gpuweb#1710 (comment)

I note that there are tools, integrated with Web Platform Tests, that process specs every 6 hours and that do the same thing:
https://github.com/w3c/webref/blob/master/ed/idl/gpuweb.idl

[tidoust]

Right, the workflow is the following:

• Reffy crawls specs every 6 hours to populate webref.
• Roughly once per week, a new version of the @webref/idl npm package is released from contents in webref. Good thing about this package is that it guarantees that the IDL it contains is valid (can be parsed) and consistent (only uses types defined by some specification, no duplicate definitions, etc.). If there is a problem with the IDL, we will create a temporary patch and raise an issue (or a pull request) in the relevant repo to have the spec fixed.
• The released version of @webref/idl npm package is then used to update interfaces extracts in Web Platform Tests. See Sync interfaces/ with @webref/idl 2.1.3 for instance.
• These IDL files are then used to run all sort of IDL tests through the IDL harness

In other words, Web Platform Tests already contains tests for the IDL defined in WebGPU, and there is a mechanism in place to maintain these tests (this may take a week or so, but all IDL updates should eventually make their way to Web Platform Tests) as well as to assess the validity of the IDL defined in WebGPU.

[kainino0x]

Is there documentation on exactly what the WPT IDL tests test?

It's impossible for these tests to be fully autogenerated, though. For example the autogenerator won't know how to create a GPUBuffer object, so it can't test copyBufferToBuffer.

[tidoust]

I realize that the end of my previous comment is at best confusing. Sorry about that. You'll still need to create IDL tests in practice. The IDL extracts in the repo just provides the raw material to ease actual testing of the interfaces but it indeed does not provide the actual tests!

The IDL harness documentation describes how to setup the IDL harness to test interfaces of interest. It does not necessarily explain what tests the IDL harness runs under the hoods. The source code of the IDL harness is obviously available, but not necessarily an easy read...

[kainino0x]

Aha, excellent, thank you for all this background info!