3

I'm integrating an enterprise application with an existing pattern.

The main problem is how to synchronize data between my system and an external system through RPC (HTTP API calls to be precise).

What I have done so far

The first thing I have done is dividing the interface in business methods such that at the beginning of the method I have a pretty clear idea of the data I will want to work with. For example the fictitious method call getIceCreamsOfColor("red") will probably make a remote call to get all the red ice creams.

getIceCreamsOfColor($color) { $remoteClient->queryForColor($color) ... }

This is generally much better than

getIceCreams($idArray) { foreach ($idArray as $iceCreamId) { $remoteClient->queryForId($iceCreamId) ... } }

(a) Inside every business method at some point I would have to spend 1 second or so to download the remote data and create local stubs of the remote objects.

(b) Then I perform my own logic. This mean possibly manipulating the local stubs. The local stubs are in reality business entities that are used to reason about the business domain and are queried by the front-end to visualize data.

(c) Finally I should spend another second or so flushing the local stub pool. This implies writing the stubs to disk for caching and writing to the remote server all the changed data.

Either action (a) or (c) are optional if for some reason I know the local data is still in sync or I don't care so much and I just want to view the remote data without modifying it.

At this point, inspecting the result of the remote call, the business method should return possible failures or an ok message.

Does this architecture sounds reasonable to you?

Do you think the local caching is dangerous/unnecessary for remote data sync?

How do you address the problem of dealing with an external service that stores your data and you want to keep in sync?

The main reason there is only 1 call to write data is that the remote API have a cap set very low.

Improve this question
6
  • Can you change the title of your question to something that is more descriptive of the specific problem you're trying to solve? (I think it's the last sentence in the body of your post). Commented Apr 29, 2016 at 14:46
  • 2
    Honestly, this looks like a local cache. Commented Apr 29, 2016 at 14:54
  • You sort of buried the lead, didn't you? Commented Apr 29, 2016 at 15:12
  • I thought that was a common problem. Commented Apr 29, 2016 at 15:14
  • Usually, programmers have access to servers that are controlled by the corporate entity that they are working for. Or, they have strategic partnerships with the companies that control said servers. Or, they simply have a subscription to the service without caps. Commented Apr 29, 2016 at 15:15

2 Answers 2

Reset to default
2

This sounds reasonable, save for two considerations.

Multiple client access: what if your service is handling two simultaneous requests to modify the same remote entity? At the very least, you have to make sure they do not update the remote side in an interleaved manner, resulting in a remote state none of them expects. You should (optimistically) lock access to distinct remote entities.

Crash resilience / consistency: what if your service crashes halfway between updating a local copy and updating the remote object? What if the remote side crashes or becomes unavailable for some time?

Basically both of the problems are about transaction control. A typical piece of software that gives you transaction control is a (relational) database; possibly caching your local state in it, and using it to sequence parallel access and replay / rollback changes after a crash o restart might make your part more predictable.

Improve this answer
1

I'd rather that the server providing the data actually performed work on it for you, rather than farm it out and let you manipulate it.

How are you going to handle multiple clients manipulating the same data at the same time ? I'm thinking particularly when you write it back - how will your service resolve who is correct ?

Furthermore, how will the service validate that you've changed/amended the data in a valid fashion ? Who determines the validity of the data being persisted.

I would normally expect that you'd use a API to ask the server to do something with that data. Consequently the server can manage concurrent requests, and perform the actions for you and persist the data back again whilst being certain that it's manipulated the data in a consistent and valid fashion. To my mind, if the server is manipulating and persisting modified data, then it owns that data, and shouldn't entrust mutations to any other service.

Improve this answer
1
  • 1
    I agree with you, unfortunately, asking the server to manipulate data would result in too many API request, the server would reject further request at a certain point. I will update the question. Commented Apr 29, 2016 at 15:03

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.