1

Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance issues and dictated by third party software.

The platform is essentially a naïve Micro-Service pattern followed by being accessed in a Multi-Tenant fashion.

We have a core API that inserts core data, but the core data needs to be complimented by other data, in this example: Assets, C1 and C2.

Now the idea is to insert core data, then publish to a queue, each module needs to subscribe, compliment the data and then publish back into the queue, which then is subscribed via the other modules sequentially. By that point all data is complimented giving a full set of data, which then is finally subscribed via the last module, which then does some super-uber magic and returned.

We expect volume of thousands of requests and possibly millions in due course. Performance/speed is absolutely vital but more performance. Limiting timeouts and hang-time, is also vital and can be achieved via parallel/background tasks. It's the throughput volume that is the highest priority.

Diagram

Questions:

  1. Is such architecture feasible?
  2. Can queuing based technology have/contain the ability to publish, subscribe the same data sequentially until that data is deemed complete? each time we receive the data compliment the data and push back i.e. publish
Improve this question
7
  • I've removed all the odd formatting. It was very distracting. Commented Mar 7, 2018 at 15:19
  • @JimmyJames I spent a lot of time on that haha - appreciated. Commented Mar 7, 2018 at 15:21
  • You are using the words publish/subscribe and queue together. This is non-standard terminology. You publish and subscribe to topics and get/put to queues. Which are you talking about here: queues or topics? Commented Mar 7, 2018 at 15:21
  • @JimmyJames queuing is new to me, my apologies. It's terminology used from research - typically rabbitMQ. So from your description, Get/Put would be correct. Commented Mar 7, 2018 at 15:24
  • The phrase "compliment the data" would imply something like "hey, nice indention". I think you mean 'decorate' or 'enhance' or 'enrich' here. Commented Mar 7, 2018 at 15:24

2 Answers 2

Reset to default
1

Distributed Queue-based solutions have a number of advantages for systems that are asynchronous especially when they are throughput focused. I would not recommend using distributed queues as part of synchronous transactions.

Pros

  • Volume can be leveled out. Processing will continue at it's maximum rate and overflow will be managed on queues.
  • If designed properly, scaling is easy. Just add new processing nodes to pull from the queues.
  • Downtime on processing nodes does not (generally) prevent new requests from being accepted.

Cons

  • Queues require memory and/or disk. If you need to retain messages when queuing is down, you must write each message to disk. IO overhead can be significant.
  • Depending on the technology, you need to make a lot of decisions up front like how much depth you will support on a queue and how large the messages can be.
  • The queuing systems I have worked with do not handle large messages efficiently. There are workarounds but they can be complicated.
  • It can be difficult to find people with experience designing and developing these solutions and also people who know how to support the infrastructure.
  • Unlike a read on a database, get on a queue is destructive. It's easy for developers to commit messages to early resulting in data loss.

I can't overstate the last con: there are so many ways messages can get lost in the shuffle and it can be very difficult to detect that it is happening. For this reason, I would highly recommend you keep a transaction ledger that at the very least records every request that comes in and it's final state. The other thing to consider is that some popular queueing systems have some deep flaws. For example, one system that I have used checks every message on a queue each time you attempt to read the next message. That means if you are falling behind on processing messages, each read gets slower and slower resulting in a downwards spiral of worsening performance. Caveat emptor. I've also worked with weird configurations that make a queue work like a stack which created really unwelcome results under heavy load.

Your Design

This seems like a pretty typical queue-based approach. You have messages coming in the front door and pass them along an assembly-line solution.

What's not clear from your diagram is whether you will have mutliple queues or you literally plan to 'push back' to the same queue each step reads from. If you are planning to use a single queue, don't! I strongly advise that you use a separate queue for each stage of the process like in the diagram below:

enter image description here

You can make it work with one queue but don't get fancy. You'll just create a bunch of problems that don't need to exist. On top of that, it makes it much harder to support. With separate queues, you can see in an instant where you are falling behind: the queue with non-zero depth is being written to faster than it's being read from. And that's one last point that I've seen people struggle with. Over the long run, your depths should all be going to zero. If you have queues that are growing over time, you are heading for trouble.

One other think I advise that isn't strictly about queuing. Avoid updates to records in the DB in your processes. Use inserts only. Updating records will create contention across your processing nodes and make the solution scaling less effective.

Improve this answer
1
  • This is blooming awesome, I really appreciate the time taken to write such a splendid answer. Thank you! Commented Mar 7, 2018 at 18:49
0

Can queuing based technology have/contain the ability to publish, subscribe the same data sequentially until that data is deemed complete?

A queue is not supposed to be a database. I'm sure you can somehow hack queues to make this work but why do that.

The simplest approach would be to create a single message queue and post each incoming work item into it. A background process takes from that queue, calls into Assets, C1 and C2 (possibly in parallel), combines the data and finally invokes the magic module with the combined data.

If you want each of Assets, C1 and C2 driven by a separate message queue the process could look like this:

You could track each work item as a database row. Then you create one message queue for each Assets, C1 and C2. You post the ID of that database row to those queues. The respective services will pick up the queue message to provide data and store it into the database row.

Each service would detect if the row is completely filled (or if an error occurred and the process must be given up). When all work is completed you somehow trigger the magic module (possibly by enqueueing into another message queue created for this purpose).

If you want Assets, C1 and C2 to now know about the existence of the central database row tracking the work then you need to invert control. Each of Assets, C1 and C2 would send a queue message to some central controller service to notify it that it's own work is completed. The central controller would check if all work is done and if so invoke the magic module.

I want to advise you to chose the simplest architecture that will work. It is very easy to create elaborate queue data flows and apply "cloud patterns" on the white board. Implementing, testing and running all of this is a lot of work. Systems like this can be very brittle.

Improve this answer
2
  • A queue is not supposed to be a database. No, but do databases typically do what the OP describes in his question? Commented Mar 7, 2018 at 15:51
  • @RobertHarvey I see no issue in making them do this. Further, my answer proposes combining queues and databases. Use the queue for queuing work and the database for storing data. Commented Mar 7, 2018 at 18:08

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.