Docs or guidance on writing additional third party fan out integrations #37

Activity

1. migmartri commented on Mar 16, 2023

   

   migmartri

   on Mar 16, 2023

   Last edited by migmartri

   Member

   More actions

   Hi @garethr!

   The short answer is no :( we don't have documentation yet and in fact there is probably some work to do first to make sure the abstractions are properly set or to create an actual plugins architecture.

   I can try to in the meantime explain what I believe would be the steps involved

   1 - Implement the actual integration code

   A integration needs to implement this interface.

   chainloop/app/controlplane/internal/integrations/integration.go

   Lines 20 to 29 in e770fae

   	type Checker interface {
   	// Validate that the integration can be registered for future calls
   	Validate(ctx context.Context) error
   	}
   	type Doer interface {
   	Do(ctx context.Context) error
   	// Validate that the integration can be performed with the parameters provided
   	Validate(ctx context.Context) error
   	}

   and you can see the only integration we have right now (Dependency-Track for SBOMs) that implements this interface here

   NOTE: OCI registries although they are depicted in our diagrams as fan-out integrations, it's implementation doesn't follow this approach for historical reasons, but it will fix with this #28

   2 - Registration logic

   2.1 - Add use-case business logic

   This means that an user can register their instance of the integration in their control-plane org. Basically this step

   To do that you'd need to add an useCase to register the new integration in the biz layer. Similar to

   chainloop/app/controlplane/internal/biz/integration.go

   Line 94 in e770fae

   func (uc *IntegrationUseCase) AddDependencyTrack(ctx context.Context, orgID, host, apiKey string, enableProjectCreation bool) (*Integration, error) {

   that code will validate and store any credentials in the secrets storage, i.e dependency-track API key, and store an entry in the DB.

   2.2 - Add API endpoint

   Next, you'll need to add an API endpoint to add your new integration.

   Similar to this one

   chainloop/app/controlplane/api/controlplane/v1/integrations.proto

   Lines 25 to 27 in e770fae

   	service IntegrationsService {
   	// ORG related CRUD
   	rpc AddDependencyTrack (AddDependencyTrackRequest) returns (AddDependencyTrackResponse);

   3 - Enable attachment to workflows

   At this point you'll have a mechanism for your users to add this new integration to their organization, so they'll be able to get smth like

   $ chainloop integration ls
   ┌──────────────────────────────────────┬──────────────────┬──────────────────────────────────────────────┬─────────────────────┐
   │ ID                                   │ KIND             │ CONFIG                                       │ CREATED AT          │
   ├──────────────────────────────────────┼──────────────────┼──────────────────────────────────────────────┼─────────────────────┤
   │ c48f3041-5745-4773-aed9-ccf2f288b2e4 │ Dependency-Track │ host: https://dependency-track.chainloop.dev │ 21 Dec 22 16:29 UTC │
   │                                      │                  │ allowAutoCreate: true                        │                     │
   └──────────────────────────────────────┴──────────────────┴──────────────────────────────────────────────┴─────────────────────┘
   

   Next, you'll need to implement how to allow your users to attach this new integration to their workflow, so it gets executed once an attestation is received.

   to do that

   3.1 - Extend API with custom configuration

   If the attachment command requires additional, specific configurations, you'll need to extend this oneof and it's handling

   chainloop/app/controlplane/api/controlplane/v1/integrations.proto

   Lines 96 to 99 in e770fae

   	message IntegrationConfig {
   	oneof config {
   	DependencyTrack dependency_track = 1;
   	}

   3.2 - Handle attach to workflow logic

   chainloop/app/controlplane/internal/biz/integration.go

   Line 269 in e770fae

   func validateAttachment(ctx context.Context, integration *Integration, credsR credentials.Reader, ic *v1.IntegrationConfig, ac *v1.IntegrationAttachmentConfig) error {

   4 - Implement the actual firing mechanism

   Now our new integration is ready to 1) be registered by any organization and 2) be attached to as many workflows as needed.

   Next, we need to implement the actual actuation of the integration code you implemented in step 1.

   And this is by far the part that needs more work. I have pending adding an issue about adding an event bus (nats.io or similar) to handle persistence, retries, etc.

   but for now the code is executed in the service implementation when the attestation is received, like for example

   chainloop/app/controlplane/internal/service/attestation.go

   Line 223 in e770fae

   go func() {

   I know that this is a lot to digest and it's current architecture is not great. As I mentioned before, it might make sense to try to encapsulate most of these changes as part of a new registration/attachment interface, so the custom code related to adding a new integration will be mostly encapsulated in the first step.

   I am going to take a look at that as part of #28 but if you have any thoughts, want to chat or help please let me know!!

   Thank you!
2. garethr commented on Mar 16, 2023

   

   garethr

   on Mar 16, 2023

   Author

   More actions

   I love "The short answer is no" and then the longer answer being all the details that would be needed, along with all of the context 🥇

   I'll definitely take a look if/when I have time, and will track the issue mentioned.

   Thanks for the detailed response.

   😄React with 😄1Reacted by Miguel Martinez Trivino
3. migmartri commented on Mar 16, 2023

   

   migmartri

   on Mar 16, 2023

   Member

   More actions

   Sounds good! and yes, I error on the verbosity side as long as a TLDR is in place haha :)

   For the record, and to elaborate on my previous statement...

   ... there is probably some work to do first to make sure the abstractions are properly set or to create an actual plugins architecture.

   These are some of the additional work streams around third-party integrations that I have on top of my priority list.

   • [fan-out integration] new SDK #38
   • [fan-out integration] add resiliency to firing mechanism #39

   In any case @garethr, if you have in mind any specific integration you'd like to explore or have added to Chainloop, feel free to let us know.

   Having an integration candidate will also help with the effort on defining the right abstractions and documentation.

   Thank you!
4. migmartri

   self-assigned this

   on May 10, 2023
5. migmartri

   pinned this issue on May 15, 2023
6. migmartri commented on Jun 21, 2023

   

   migmartri

   on Jun 21, 2023

   Last edited by migmartri

   Member

   More actions

   Hi @garethr. We just wanted to let you know that we've put in place a new plugins sdk and documentation. This allows anybody to add a fan-out integration in an encapsulated manner. Note: plugins are not fully decoupled from there core yet, we will get there :) #195

   We also expect to have a build a plugin example #204 published sometime in the future, but in the meantime, is there any integration endpoint/use-case that you could think it would be interesting to implement? We'd happy to give it a crack.
7. migmartri

   unpinned this issue on Jul 20, 2023
8. migmartri

   closed this as completedon Mar 11, 2024