Add support for AsyncAPI v3

[thompson-tomo]

I want to be able to read my AsyncAPI v3 document and get back an asynci api document.

[VisualBean]

This is already on the roadmap, its more an issue of time, rather than anything else at this point.

[thompson-tomo]

That's great, let me know if there is anything i can do to help the process along. :)

[VisualBean]

That's great, let me know if there is anything i can do to help the process along. :)

I'll try to break down what needs to happen. Perhaps we can divide and conquer

[thompson-tomo]

Thanks, I have also raised a community discussion (https://github.com/orgs/asyncapi/discussions/1140) about standards & conventions going forward. I see v3 as a good opportunity to lay the right foundation.

[ByronMayne]

I was taking a crack at this and trying to break it up into phases. The first step was trying to make a Migration system for the Json Nodes. Take in the json node tree and apply transformations to migrate them up or down. This would work like the JS tool asyncapi/converter-js. This would allow the library to read 3.0.0 but users would still only be able to consume the 2.6.0 models. This gets us a step in the right direction.

The next part is more annoying but also I think it allows the project to be cleaned up a bit. I would suggest the following to be done.

• Move the readers and their helpers from AsyncAPI.Readers into AsyncAPI.
• Move the models and deserializers from from AsycnAPI.Reader into it's own library AsyncApi.Models.2
• Add migrations between versions inside of AsyncApi.
• Move the writers from AsyncApi into AsyncApi.Models.2.
• Create new library AsycnApi.Models.3.

The idea would be that the base AsyncApiLibrary would have the helper types, readers and base writer classes defined. Then each of the model libraries could add extension methods for reading and righting to their own version.

using LEGO.AsyncApi;
using LEGO.AsyncApi.Models.2;
using LEGO.AsyncApi.Models.3;
using AsyncApiDocument2 = LEGO.AsyncApi.Models.2.AsyncApiDocument;
using AsyncApiDocument3 = LEGO.AsyncApi.Models.3AsyncApiDocument;

public static class Example 
{
   public static async Task Writing()
   {
       AsyncApiWriter writer = new AsyncApiWriter();
       AsyncApiDocument2  doc2= new AsyncApiDocument2 ();
       AsyncApiDocument3 doc3 = new AsyncApiDocument3 ();

       // I doubt we would need this generic extensions since at the end of the day we are just saving json/yaml 
        await writer.WriteAsync(doc2);  // Extension method added by Models.2
        await writer.WriteAsync(doc3 ); // Extension method added by Models.2
   }
   
   public static void Read(Stream stream) 
   {
         AsyncApiReader reader = new AsyncApiReader();

         // Generic read
         AsyncApiDocumentBase baseDic = await reader.ReadAsync(stream);
         // Explicit read 
         AsyncApiDocument2  doc2 = awaitReader.ReadV2Async(stream); // extension method  by Models.2
         AsyncApiDocument3  doc3 = awaitReader.ReadV3Async(stream); // extension method by Models.3
    }
}

To do the GenericRead version we would have to do a plugin model and switch on the Version property in the doc.

There are other directions that can be taken as well, this was just my 2c

[thompson-tomo]

@ByronMayne yes i am thinking of a similiar approach and i would propose we use a disucssion topic to develop plans etc hence creating https://github.com/orgs/asyncapi/discussions/1140

[VisualBean]

I think it would be great to move reading and writing into the same project.
Effectively removing the readers.

So the api becomes more along the lines of a static factory like AsyncApiDocument.Load or .Parse.

There is no real need for a complete reader api - it's not like we do any sort of memory efficiency patterns in them anyway.

For V3;

I firmly believe we should 'upgrade' V2 to V3 during reading. - I don't think there are any issues between V2 and V3 standing in the way of this as the main difference hinges on how things can reference eachother (which we can infer for V2).

One thing I don't want, is to push figuring out which asyncapi doc version some Json stream is, to the lib consumer.
Why I am pushing for 'upgrading' to V3 models. There is no need to keep the V2 models around as I see it.

So my plan is this;

1. Change current models to reflect V3.
2. Refactor SerializeAsV2 to output V2 from V3 models.
3. Refactor current V2 derializers to fill V3 models
4. Add new SerializeAsV3 method to the interface.
5. Make new deserializers for V3.
6. Think about load/parse api.

I am currently on parental leave, so my responses might be somewhat slow

[VisualBean]

Thanks, I have also raised a community discussion (https://github.com/orgs/asyncapi/discussions/1140) about standards & conventions going forward. I see v3 as a good opportunity to lay the right foundation.

Note that we recently started a working group for this sort of work.
I'll find the link when I get near a computer 😅

Essential building blocks
https://github.com/orgs/asyncapi/discussions/1128

There is a problem with api design for keeping models vs reading and writing however - so Im not sure this is such a great idea.

As those models would either be simple Pocos with no logic leaving problems on the consumer side.

Or have some logic and internals, which hinders proper (de)serialization.
Unless ofcourse it's developed proper. But that takes a lot of effort and foresight.

One thing that comes to mind is the securitySchemaDefinition, which here is implemented as a custom dictionary.

Another is the JsonAny types, which used to be implemented as custom models, but I've moved away from that to make the usage easier on library consumers.

Many of these things could not happen if the models came from somewhere else.

[ByronMayne]

One question is should the library only ever support the 'latest' version of the models or should there be C# objects for each version of the api. For example keeping the existing models and making whole new ones for v3.

[ByronMayne]

For the migration of the from 2.6.0 -> 3.0.0 I was working on this. https://github.com/ByronMayne/AsyncAPI.NET/blob/feature/api-v3-migration/src/LEGO.AsyncAPI/Serialization/MigrationV3.cs. It's not done yet but the idea is to mutate the json date before the models read it. The way it's written now we can also read/write any version of the schema.

The first goal would be downgrade a 3.0.0 spec to our current models, then when the 3.0.0 models are ready it should just work out of the box.

[VisualBean]

One question is should the library only ever support the 'latest' version of the models or should there be C# objects for each version of the api. For example keeping the existing models and making whole new ones for v3.

I only see one version needed for the models. From a code point of view they represent the exact same thing. However having multiple models creates a weird interface for the library, pushing to the consumers to code against an unsupported spec version.

The whole library was architected towards this idea of only ever having one version of the models, with serialization and deserialization being version specific, so to speak.

For the migration stuff, see the plan I made, it should work nicely simply refactoring existing V2 deserializers to fill the new models (there is a need for more reference stuff however) and it should be easily pluggable adding V3 deserializers

[VisualBean]

@ByronMayne @thompson-tomo
I am almost ready to get started on the work for V3, I am finishing up some work on implementing Avro Schemas, and then ill get cracking on the v3 support.

[ByronMayne]

Hey @VisualBean,

That is awesome! If there is an area you would like help on let me know.

[VisualBean]

Update to this issue:

I finally managed to finish a needed update to how references work.
This was needed for V3 to run smoothly with the differences between V2 and V3.
Now that this is FINALLY out of the way - I will dedicate around 4 hours per week, for doing the upgrade.
90% of the code will just be copy paste, so it really shouldnt be all that difficult.
The difficult bits is the upgrade logic from 2 -> 3, due to some differences in logic. However as with all things code.
We are gods in this universe that we ourselves have built.

Sorry for the delay.

Edit: If you are curious about the update to references, check out the Beta release (or vnext branch)

[jrsearles]

Any chance you can release a beta with v3 support on nuget? I tried the current 6.x beta, but it did not appear to have v3 support.