Proxy docs outdated, should point to fluentd instead

[Sheeri]

https://github.com/OpenLineage/docs/blob/main/docs/development/ol-proxy.md mentions the old proxy.

[dolfinus]

Do you mean something like this?

[INPUT]
    Name http
    Port 9200
    Mem_Buf_Limit 50MB
    successful_response_code 201
    success_header Content-Type application/json

[OUTPUT]
    Name http
    Match *
    host marquez-host
    port 5000
    uri /api/v1/lineage

https://docs.fluentbit.io/manual/pipeline/inputs/http
https://docs.fluentbit.io/manual/pipeline/outputs/http

[Sheeri]

No, I mean that https://github.com/OpenLineage/docs/blob/main/docs/development/ol-proxy.md contains instructions for how to build a proxy. At least some of this is outdated, for example:

Not only the OpenLineage proxy is useful in receiving the monitoring the OpenLineage events, it can also be used to relay the events to other endpoints. Please see the example of how to set the proxy to relay the events via Kafka topic or HTTP endpoint.

Relaying events to endpoints is generally accepted to be best done by fluentd. I'm not sure if a proxy is still favorable for receiving/monitoring OpenLineage events in general, but if so, then most of the page should be rewritten.

[dolfinus]

My question was about fluentd configuration which should be compatible with OpenLineage clients. In the previous comment I've added an example of config for fluent-bit, which is quite similar to fluentd.

[dolfinus]

Ah, I see, this is about this fluentd plugin:
https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd

I thought this is all about some generic fluentd input, like http or tcp.

[Sheeri]

Right! Because fluentd is preferred as a proxy, what purpose does this "ol-proxy" serve?

[dolfinus]

Whell, for now README of fluentd plugin says:

Fluentd support is experimental and could be changed or removed in a future release.

[Sheeri]

If you know this page is accurate then you can close this issue. I don't actually know which is why I am asking. Sent from Proton Mail mobile

…

-------- Original Message --------

On Apr 3, 2024, 4:17 PM, Maxim Martynov wrote: Whell, Readme of fluentd plugin says: > Fluentd support is experimental and could be changed or removed in a future release. — Reply to this email directly, [view it on GitHub](#293 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AANPQ2DFVI3LBDHN3RV76KLY3RPWZAVCNFSM6AAAAABD6PM4AOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVGUYDGNJRGY). You are receiving this because you authored the thread.Message ID: ***@***.***>

[pawel-big-lebowski]

README content of Fluentd proxy should be moved to https://openlineage.io/docs which is https://github.com/OpenLineage/docs repo. Perhaps we can leave this issue not to forget this.