2

Introduction

Hi everyone,

in my company we are using microservice approach and of course are trying to do it as correct as possible. There is a new requirement coming up where I have laid out a proposal for an architectural design. I have received feedback to change the implementation, however this would require direct HTTP connection over 2 microservices. I wanted to ask if there are cases where chatty microservices are okay and maybe in my example we are not even talking about chatty microservices

Context

So, we want to GET and POST data to an external API that is outside our company. This API updates their data once a day so the idea was to use a TimerTrigger (Azure specific, like a cronjob) that synchronizes the data from the GET endpoints once a day.
As I mentioned we would also do some POST requests where we send data the user has created to their API. For that part I thought we put the user requests into a queue that a data synchronization service can read from and process whenever the time is there.

I will go into more detail now to first talk about my approach and then talk about the second approach that was suggested by our architects

Proposal one (My initial plan)

My plan is to have these things

  • Microservice 1 (ApiService from now on) for our internal end-users without any connection to the external API
  • Microservice 2 (DataSynchronizationService) for synchronizing data from the external API to our system to be consumed
  • CosmosDb to store the synchronized data
  • ServiceBus to use POST endpoints in an asynchronous way

So basically, we will have one ApiService which is responsible for allowing OUR users to retrieve data from. This ApiService WILL NOT have any direct connection to the external API but use the data that is synced by the DataSynchronizationService.

So basically, an end-user makes a request RETRIEVE DATA to the ApiService which takes all the data from the database without relying on the external api. The DataSynchronizationService will run once a day to update the data in the database.

When an end-user makes a request to POST DATA to the ApiService, the ApiService will add an item to the Queue which the DataSynchronizationService will pick up and send to the external API

However, I already have some points I'm not so sure about with this approach

  1. Is it okay that 2 microservices are sharing the db? I could create an EventHub in between these two but for me this feels like way too overengineered and duplicated as the one MicroService is just for DataSynchronization
  2. Should maybe all of it happen in ONE microservice, because everything is coupled together? But then I would send a queue to the same service just to run it in the background?

Proposal two (Suggested plan)

For the suggested plan we will have

  • Microservice 1 (ApiService from now on) for our internal end-users which handles anything like synchronizing the data, but also providing it for the user
  • Microservice 2 (DataSynchronizationService) which will only contain the cronjob and service bus
  • CosmosDb to store the synchronized data
  • ServiceBus to use POST endpoints in an asynchronous way

Basically their idea is the following

The ApiService will handle everything. The user will query the ApiService to retrieve the data, but also to create the data. The ApiService will also have an endpoint to synchronize the data. This endpoint is called by the DataSynchronizationService. So the DataSynchronizationService has just a TimerTrigger (Azure function like cronjob) that does nothing else but make an HTTP request to our API. For me, this already feels like an anti-pattern because the DataSynchhronizationService makes a http request to the ApiService which makes again multiple http requests to external-service.

Then, when ApiService will create a Queue Item to process the request of the user, the DataSynchronizationService will read the Queue and again make an HTTP Request to the ApiService which will in turn again make a request to the external-service.

For me, this brings multiple problems

  • We are coupling the two services with multiple HTTP requests
  • We build a queue to write in, just to read it from another service, just for triggering the initial service who created the message again
  • We are creating a second microservice just for handling non-http specific things like ServiceBus and TimerTrigger

Maybe both solutions are wrong? Because I could also imagine that everything belongs together, so all of it should be one microservice? However in my proposal I thought about splitting it by

  • One microservice for end-user communication
  • One microservice for letting our system use data of external data

I would love to get your insights

Improve this question
11
  • I'm sorry if I missed this but what should happen if data that hasn't been created and/or synced on the external system is requested from your API. Commented Sep 14, 2023 at 17:44
  • No worries, you didn't miss it as I didn't explain that case. If our system doesn't have the data I would say this would be a 404. Of course we could, as a fallback check the external API as well, but I would say we keep it so that the system only knows about synced data and that's the only truth. Would this change the solutions? Commented Sep 14, 2023 at 18:50
  • I guess I'm confused about what is shared here. If updates are only being sent to the external provider and they only way they are 'seen' on lookup is from the daily snycing, why is there any need to share a DB? Maybe I'm misunderstanding the question. Commented Sep 14, 2023 at 18:56
  • In the first scenario the shared db would be, because you would have one service which syncs the data from the external provider to a local db and another service which is used by end-users to retrieve that data Commented Sep 14, 2023 at 19:07
  • additionally if the end-user makes a post request to the one service we would add a message to the queue for the data synchronization service to create this object in the external api as well Commented Sep 14, 2023 at 19:07

2 Answers 2

Reset to default
2

As you identified yourself, the two services are very coupled together and that holds for both designs. Either they cannot be freely developed independently due to the shared database, or there is a strong temporal coupling that causes the synchronization to go wrong if the two services can't communicate with each other at the wrong moment.

As you identified yourself, it should be one microservice that is responsible for maintaining a synchronized local view on that data from the external party.
One microservice means that it is maintained, versioned and presented to the outside world as one unit, but it is nowhere written that one microservice must also be deployed as a single executable. If your infrastructure makes it easier to deploy a microservice in two parts (one to listen to HTTP requests and another one to be executed by a scheduler), then that is an entirely valid deployment strategy. And if you want to use a Queue to communicate between those parts of your microservice, that is an implementation detail that nobody but the team working on the service should worry about.

So, I would go with your proposal, but call it one microservice that happens to be deployed in two parts.

Improve this answer
5
  • Hi. Thank you for the answer. Would the second proposal not be also valid then if you see it as one unit split into two parts? Or would the direct http communication between them make it not one unit anymore? In the second proposal the second service would be just for orchestrating, like on a cronjob call the other api via HTTP. Would you think this would violate it and still go with proposal one? Thanks again for your answer! :) Commented Sep 15, 2023 at 7:19
  • @Musterknabe, The second proposal could also be described as a single service in two parts, but I don't like the roundabout route that especially the POST data takes in the proposal. The aim there seems to be to ensure that only a single service accesses the database and that the interaction with the external API is in a single service. But that argumentation breaks when you consider the TimeTrigger part to be part of the same microservice. Commented Sep 15, 2023 at 7:28
  • I think the main reasoning from the architects was that azure functions are super easy for using triggers like TimerTrigger for example, however if they are not durable functions (azure specific thing) then they cannot be scaled easily in k8s, while the app service (regular service) can be scaled easily and if we do all the data synchronization in an azure function using the trigger then it will be harder. BUT I think that this case does not apply to us as the data will be synced like once a day MAXIMUM so I don't think we even need to scale up. Commented Sep 15, 2023 at 7:35
  • Even for the POST which can happen multiple times a day I don't think we will run into an issue scaling. And as far as I understood the scaling is just an issue if using TimerTrigger because then it's not acting as a load balancer but just as two instances. However for a ServiceBus each message is only consumed once so scaling up there should be working correctly and not act as two independent instances. Commented Sep 15, 2023 at 7:36
  • Thank you, I will bring it forfward like this in the architects meeting and come back, hopefully latest today and will approve the answer Commented Sep 15, 2023 at 7:37
1

Is it okay that 2 microservices are sharing the db?

It can be perfectly okay - although whether you can then still call them legit microservices is up for debate. It's a tradeoff that depends on your context: what is the planned db load in the future, how much horizontal scalability do you need (a single db could hamper it), could the database schema be a contention point between 2 dev teams, etc.

Should maybe all of it happen in ONE microservice, because everything is coupled together? But then I would send a queue to the same service just to run it in the background?

I wouldn't recommend it, as the features from microservice 2 will be highly dependent on the external system and you probably don't want to redeploy everything at every small change in their API. The reasons for your main API to change will mostly be different from the integration microservice's reasons to change. A separate service also brings robustness and fault tolerance, as one can crash or malfunction without the other being affected if appropriate communication buffers exist between them.

We are coupling the two services with multiple HTTP requests

Not necessarily a big deal if it stays at that, but the double coupling you raise in your second bullet point indeed looks like overkill. You basically get the complexity of a message queue without the benefits.

However in my proposal I thought about splitting it by

One microservice for end-user communication

One microservice for letting our system use data of external data

I'm not sure about the term "microservices" to describe that ecosystem, and I'm also not sure how this differs from Proposal one. But I've implemented similar solutions to similar integration problems in the past and it works well.

Improve this answer
6
  • Hi thanks for the answer. I think the point about features from microservice 2 to be highly dependent on external api and not needing to redeploy it is a really good argument for splitting it. Thank you! :) Commented Sep 15, 2023 at 9:25
  • Regarding the second bullet point it could also be done doing Hangfire (background tasks for .NET) so we wouldn't necessarily need a queue, however it could not be scaled up individually from the others and the point regarding changes from external api still stand. What do you mean by your last point that you are not sure about microservice for that ecosystem? How would you describe it and how should implementation be done to fulfill the naming of a microservice Commented Sep 15, 2023 at 9:33
  • The fact that your service might not be limited to a small slice of business functionality and not having a separate small team taking care of it might be signs. Also sharing the same database would hurt independent deployability. See microservices.io/index.html for basic properties of a microservice. Note that it not being a microservice is completely fine though. It's not something you should absolutely seek. Commented Sep 15, 2023 at 11:19
  • 1
    Hm, I don't think that each microservice has to be done by a separte team to be a microservice as not all services will be actively developed, so in our company a team has multiple microservices under the hood. We have different teams and each of them has a set of microservices assigned to it. The page you shared actually has a specific section for shared databases microservices.io/patterns/data/shared-database.html which is where I got the initial idea from Commented Sep 15, 2023 at 11:40
  • 1
    No worries, I didn't want to throw shade or anything. It's def. an open topic that companies interpret differently. It's the same with SCRUM, almost every company does it different and you have to find the perfect balance that fits with your use case. I will go with solution 1. Thank you for your input! Commented Sep 15, 2023 at 12:05

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.