Skip to main content
Software Engineering

Return to Question

deleted 1 character in body
Source Link
Abhishek
Abhishek
  • 129
  • 3

There are 60X Million Shipments per day. Each shipment has about 50 metrics to be calculated. Each metric is calculated based on a type of the event(Let's say event_1 has the required information to calculate metric_1, event_2 .. metric_2 and so on). All the events are independent of each other apart from one dependency, a single event(let's say event_1) which has vital information required to process each of the other events.

The current design:

(In Order)Scenario 1:event_1 arrives first, we calculate metric_1 and store the vital information required to process other events in DynamoDB. Other events(event_4, event_2,....) arrive and are processed by accessing the information from DynamoDB.

(Out of Order)Scenario 2: event_3 arrives first, system checks for required information in DynamoDB and fails, the system places the event in the dead letter queue to be retried after a period of time. One event_1 arrives and is processed, the other events go through.

Is using a data store and retry mechanism the right approach to resolve the dependency on the base event(event_1)?

Are there better approaches/patterns to solve the event dependency problem?

Additional Context: Although I believe this information is irrelevant, I am giving it anyway if it helps. Source of Events: SNS topics, Event Processing: SNS->SQS->Lambda, Data Store: DynamoDB, Metrics are stored in RedShift.

There are 60 Million Shipments per day. Each shipment has about 50 metrics to be calculated. Each metric is calculated based on a type of the event(Let's say event_1 has the required information to calculate metric_1, event_2 .. metric_2 and so on). All the events are independent of each other apart from one dependency, a single event(let's say event_1) which has vital information required to process each of the other events.

The current design:

(In Order)Scenario 1:event_1 arrives first, we calculate metric_1 and store the vital information required to process other events in DynamoDB. Other events(event_4, event_2,....) arrive and are processed by accessing the information from DynamoDB.

(Out of Order)Scenario 2: event_3 arrives first, system checks for required information in DynamoDB and fails, the system places the event in the dead letter queue to be retried after a period of time. One event_1 arrives and is processed, the other events go through.

Is using a data store and retry mechanism the right approach to resolve the dependency on the base event(event_1)?

Are there better approaches/patterns to solve the event dependency problem?

Additional Context: Although I believe this information is irrelevant, I am giving it anyway if it helps. Source of Events: SNS topics, Event Processing: SNS->SQS->Lambda, Data Store: DynamoDB, Metrics are stored in RedShift.

There are X Million Shipments per day. Each shipment has about 50 metrics to be calculated. Each metric is calculated based on a type of the event(Let's say event_1 has the required information to calculate metric_1, event_2 .. metric_2 and so on). All the events are independent of each other apart from one dependency, a single event(let's say event_1) which has vital information required to process each of the other events.

The current design:

(In Order)Scenario 1:event_1 arrives first, we calculate metric_1 and store the vital information required to process other events in DynamoDB. Other events(event_4, event_2,....) arrive and are processed by accessing the information from DynamoDB.

(Out of Order)Scenario 2: event_3 arrives first, system checks for required information in DynamoDB and fails, the system places the event in the dead letter queue to be retried after a period of time. One event_1 arrives and is processed, the other events go through.

Is using a data store and retry mechanism the right approach to resolve the dependency on the base event(event_1)?

Are there better approaches/patterns to solve the event dependency problem?

Additional Context: Although I believe this information is irrelevant, I am giving it anyway if it helps. Source of Events: SNS topics, Event Processing: SNS->SQS->Lambda, Data Store: DynamoDB, Metrics are stored in RedShift.

Source Link
Abhishek
Abhishek
  • 129
  • 3

Solving Event Dependence in Event Driven Systems

There are 60 Million Shipments per day. Each shipment has about 50 metrics to be calculated. Each metric is calculated based on a type of the event(Let's say event_1 has the required information to calculate metric_1, event_2 .. metric_2 and so on). All the events are independent of each other apart from one dependency, a single event(let's say event_1) which has vital information required to process each of the other events.

The current design:

(In Order)Scenario 1:event_1 arrives first, we calculate metric_1 and store the vital information required to process other events in DynamoDB. Other events(event_4, event_2,....) arrive and are processed by accessing the information from DynamoDB.

(Out of Order)Scenario 2: event_3 arrives first, system checks for required information in DynamoDB and fails, the system places the event in the dead letter queue to be retried after a period of time. One event_1 arrives and is processed, the other events go through.

Is using a data store and retry mechanism the right approach to resolve the dependency on the base event(event_1)?

Are there better approaches/patterns to solve the event dependency problem?

Additional Context: Although I believe this information is irrelevant, I am giving it anyway if it helps. Source of Events: SNS topics, Event Processing: SNS->SQS->Lambda, Data Store: DynamoDB, Metrics are stored in RedShift.