0

At my company we are revising some backend architecture. I think I have identified a use case where event streaming (for example with Apache Kafka or RabbitMQ) makes sense.

Because me (and everyone else- at my company) are pretty much noobs in that space, I'd like to validate my idea.

Business flow and prerequisites

We have a bunch of Microservices that are responsible for importing an encrypted document from an external source.

  • One for downloading it and saving it, acting as a sort of archive
  • One for decrypting it
  • Some others for associating the document with other business objects

The flow is roughly as follows(each color is a different microservice):
enter image description here

Current situation and problem
All communication happens with REST APIs, exposed by the microservices over HTTP. This was very simple to implement but has since revealed a big problem: If one service in that chain is down, the import is interrupted and can't be continued without manual intervention.

Proposed idea
I'd like to replace the direct communication over HTTP with message queues between the different services, like that:
enter image description here

This way the service that downloads the document could just push an event each time it downloaded a document. If the decryption service is online, it will decrypt it right away. If not, it will happen start the moment it is online again.

Similar for the other services, where the decryption service sends an event it has finished decrypting a document.

So my question(s) are:

  • Is this a good use for events?
  • Are there obvious tings to improve on this design?
  • Are there any design patterns that I should study or look up?

Thanks in advance for any answers and cheers :)

Improve this question
11
  • If one service in that chain is down or the queue is down or messages are stealth or gone. Isn't it? Why do all three stages have to "happen" at different moments? Commented Dec 7, 2022 at 14:41
  • @Laiv The rational was that the download should be independent from decrypting because the file is only available for a short duration of time. Thus even if the decrypting doesn't work, we have a copy available and save. But I see your point that message queues are only shifting the problem arround. Commented Dec 7, 2022 at 15:30
  • I was about to say that it was overkill, but maybe I was missing something important. For example that decryption can only happen when "someone" provides the secret or the certificate, etc. Perhaps, it's easier to do an ETL based on folders and files. Download folder, decrypted folder, processed folder. So business and decrypt services just check their respective folders' content periodically. Commented Dec 7, 2022 at 15:51
  • To add to @Laiv 's comments... His idea really is pretty good. If you're using a tool like Spring Integration to manage your events, you can plug-in different event handlers over time and your actual services that process the files will not know where those events are coming from. For instance, the events could come from Kafka or ActiveMQ or a directory watching service that checks to see if there are files present and then it will fire the events itself. Commented Dec 7, 2022 at 16:26
  • It also adds persistence. Files are there to be processed unless they are moved. It just takes to move a file to the right directory to "solve" or "complete" a faulty process. No message lose is possible. Commented Dec 7, 2022 at 16:30

1 Answer 1

Reset to default
1

After reading the question a couple of times, I concluded that either through direct calls between services or messaging, both solutions cause the same problems. I'm talking about problems related to distributed computing.

If one service in that chain is down, the import is interrupted and can't be continued without manual intervention.

If the queue goes down, more of the same.

The messaging introduces different problems too, for example, race conditions. But it's not the only: message delivery policies, security, filtering, routing or acknowledgement of receipt to mention some.

All in all, even if you could deal successfully with all these things, I still would do the same question. Why so complicated? Why the file downloading, decryption and processing happen at different times? Isn't one of the most common ETL?.

So my answer is a question. Wouldn't it be simpler to make a workflow based on folders where services look for files periodically?

The idea is simple:

  • Service A downloads the file into /downloads/encrypted
  • Service B looks for encrypted files in /downloads/encrypted
    • For each file, it makes a decrypted copy in /downloads/decrypted
    • Moves encrypted files decrypted successfully to a different folder
    • Moves encrypted files not decrypted successfully to a different folder
  • Service C looks for decrypted files in /downloads/decrypted
    • Process the files
    • Moves files processed successfully to a different folder
    • Moves files not processed successfully to a different folder
  • Service D looks for faulty files to do something with them.

Service A is highly scalable. We can have many threads or replicas downloading files as we need. Service B and C can be scalable too, but we need to lock files to avoid decrypting and processing the same file twice or more.

Retrying any of the jobs then takes throwing files in one or another folder. Checking the state of the pending jobs can be as simple as looking into the folders.

The file system has features we take advantage of. For example, security, meta-data and persistence.

The file system doesn't have to be shared either. Services ask the other services for new files. For example, C asks B for new files, and so does B to A. They download files into their respective file systems and do the same job.

Note that, the solution is no longer based on pushing but pulling job.

Improve this answer
8
  • What happens if the filesystem goes down? What makes the filesystem different from a queue? Commented Dec 9, 2022 at 12:54
  • 1
    Nothing happens. No file can be downloaded, no file can be read and processed. No job is lost. Even if it goes down files persist. if the system goes down while files are downloaded, the download fails and the job is marked as KO so the service can retry the download in the next iteration. If it goes down while moving files to "done" folders, the next iteration will know that those files have been processed and the state of the process so they can move'em now. Not a big problem to be honest. Given the same trade-offs, I choose the simplest solution possible. Commented Dec 9, 2022 at 13:06
  • And if the disk crashes? You need a distributed filesystem/object store to make those guarantees. And a distributed persistent queue is simpler than a distributed filesystem. Commented Dec 9, 2022 at 13:09
  • Filesystem backups and replicas are, probably, the cheapest and simplest among all the backups possible- Commented Dec 9, 2022 at 13:10
  • Now before you keep going down tho rabbit hole, I make you another question... What if Earth is struck by a jet of Gamma rays? how queues handle that? Commented Dec 9, 2022 at 13:11

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.