Writing match log to kafka is not robust #11

Activity

1. greensheng commented on Oct 24, 2019

   

   greensheng

   on Oct 24, 2019

   Member

   More actions

   We have considered the case of process crash. Through the snapshot mechanism to ensure that any log will not be lost, when the process is restarted, a snapshot recovery will be done. Therefore, there will be no data loss, but duplicate logs will be generated.
2. yihuang commented on Oct 24, 2019

   

   yihuang

   on Oct 24, 2019

   Author

   More actions

   Thanks for your reply and work. Let me elaborate a bit:
   1、Do you ensure that when snapshotting, all the messages produced to kafka get acked?
   2、AFAIK kafka producer doesn't guarantee message order, does that impact the following processes.
3. greensheng commented on Oct 24, 2019

   

   greensheng

   on Oct 24, 2019

   Last edited by greensheng

   Member

   More actions

   Thanks for your reply and work. Let me elaborate a bit:
   1、Do you ensure that when snapshotting, all the messages produced to kafka get acked?
   2、AFAIK kafka producer doesn't guarantee message order, does that impact the following processes.

   1，Yes, We use synchronous write to submit logs, writing 1000 logs at a time.When we save the snapshot, we will first check whether the snapshot.logSeq has been submitted to Kafka.
   2，Only one partition is used for each log topic, so the sequence can be guaranteed.
4. yihuang commented on Oct 24, 2019

   

   yihuang

   on Oct 24, 2019

   Author

   More actions

   2, The retry behavior in the producer side will change the order of messages.
5. greensheng commented on Oct 24, 2019

   

   greensheng

   on Oct 24, 2019

   Member

   More actions

   2, The retry behavior in the producer side will change the order of messages.

   Log producer is synchronous and single thread, and there will be no concurrent submission.
6. yihuang commented on Oct 24, 2019

   

   yihuang

   on Oct 24, 2019

   Author

   More actions

   https://stackoverflow.com/questions/49007592/kafka-message-ordering-in-partition-while-producer-retry
7. greensheng commented on Oct 25, 2019

   

   greensheng

   on Oct 25, 2019

   Last edited by greensheng

   Member

   More actions

   I see what you mean. I read the source code of Kafka go. Retrying really leads to the order not being guaranteed.

   At present, if the log consumer finds the log.seq discontinuity, it will terminate the process.

   Do you think enable. idempotence can solve this problem?
8. yihuang commented on Oct 25, 2019

   

   yihuang

   on Oct 25, 2019

   Author

   More actions

   Yes, enable.idempotence = true suppose to solve this, if kafka go is a binding to librdkafka.
9. greensheng

   closed this as completedon Nov 1, 2019