Skip to main content
Software Engineering

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

7
  • Thanks for the suggestion. I realized I missed some key information in my post so I updated it with clearer description. I hope you'll find the problems interesting. Commented Oct 7, 2021 at 18:04
  • As for your suggestion, I think the non-distributed DB might be a critical point of failure. I.e. if for some reason, I cannot insert into this DB from the 1st queue, I won't have events available in the 2nd queue for retry. In addition, while the events are stuck on the 1st queue, I also cannot query to find out colliding booking code. Commented Oct 7, 2021 at 18:06
  • im not up on kafka, but it sounds like the "compacted topic" is essentially a non-distributed db. If you cant process the first queue then its no different from any other queue processing failure, the message remains on the queue, or you push it somewhere else. Commented Oct 7, 2021 at 18:16
  • The difference is if I can send the event into Kafka, it means Kafka is running. And the Event Sourcing System, the one who sent the event (i.e. running) is also the one who's receiving the stream back. So there's no additional point of failure. Who should be up and running is up and running. If Kafka fails, the process fails early and client is notified of the failure. If Kafka acknowledges and the event get streamed into multiple nodes as state store, the node which receives a request can do validation in-memory. Commented Oct 7, 2021 at 18:20
  • 1
    i think that's a bit of a false premise, surely there are error types where you compacted topic can be broken while other queues are working in a Kafka cluster. You have an extra point of failure per working part, regardless of the technology used. Commented Oct 7, 2021 at 18:25