Timeline for How do we maintain consistent-read promise to clients + handling ID collision when using a fallback queue?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 7, 2021 at 18:36 | comment | added | JamesBoyZ | Thanks for the comments. They help me expand my thinking a lot :) | |
| Oct 7, 2021 at 18:35 | comment | added | JamesBoyZ | I agree with your statement. That's why I wanted to replace the Q with Kafka topic to maintain the number of components in the design as 2. Please forget the term compacted. It's no different from a normal topic. I picked Kafka becoz' it support stream to maintain an in-memory store. The only additional failure point I can think of is the stream where new events are streamed out a bit slower for some reasons or Kafka died right after receiving an event and before streaming out. | |
| Oct 7, 2021 at 18:25 | comment | added | Ewan | i think that's a bit of a false premise, surely there are error types where you compacted topic can be broken while other queues are working in a Kafka cluster. You have an extra point of failure per working part, regardless of the technology used. | |
| Oct 7, 2021 at 18:21 | history | edited | Ewan | CC BY-SA 4.0 | added 329 characters in body |
| Oct 7, 2021 at 18:20 | comment | added | JamesBoyZ | The difference is if I can send the event into Kafka, it means Kafka is running. And the Event Sourcing System, the one who sent the event (i.e. running) is also the one who's receiving the stream back. So there's no additional point of failure. Who should be up and running is up and running. If Kafka fails, the process fails early and client is notified of the failure. If Kafka acknowledges and the event get streamed into multiple nodes as state store, the node which receives a request can do validation in-memory. | |
| Oct 7, 2021 at 18:16 | comment | added | Ewan | im not up on kafka, but it sounds like the "compacted topic" is essentially a non-distributed db. If you cant process the first queue then its no different from any other queue processing failure, the message remains on the queue, or you push it somewhere else. | |
| Oct 7, 2021 at 18:06 | comment | added | JamesBoyZ | As for your suggestion, I think the non-distributed DB might be a critical point of failure. I.e. if for some reason, I cannot insert into this DB from the 1st queue, I won't have events available in the 2nd queue for retry. In addition, while the events are stuck on the 1st queue, I also cannot query to find out colliding booking code. | |
| Oct 7, 2021 at 18:04 | comment | added | JamesBoyZ | Thanks for the suggestion. I realized I missed some key information in my post so I updated it with clearer description. I hope you'll find the problems interesting. | |
| Oct 7, 2021 at 12:56 | history | answered | Ewan | CC BY-SA 4.0 |