2

I am working on an application that involves alerts as well as notification policies for multiple users. I am using microservices and am trying to keep them stateless if possible. I have one service that will do the alerting and another that stores and handles policies.

I currently have it partially working by the alerting service gets a message from an SQS queue (that is produced by another service when something fails). My alerting service then asks the policy service what the policy is that is attached to the message coming from the alerts queue.

This policy then contains the number of people attached to the policy as well as the user ID's of the users who are part of the policy and who should be alerted.

Then I loop over the user ID's looking up their specific notification settings. These are then stored into an array.

Part I need to figure out:

I need to figure out how to take the notification setting then alert the correct user via the proper notification method (found in their notification settings I mentioned before). This means that I want to go through each user and notify the correct user using their notification preferences but while not breaking the rules set out in the policy, ie the escalate_time as well as making sure the order of users alerted is correct. But I also want to escalate to the next user in line once escalate_time passes for each user.

I was thinking I could add a date/time stamp to the alert but not sure exactly how I would handle a user not 'resolving' an alert.

I have not done the resolving method yet because I am stuck on this alerting part and want to design both parts to work together. I expect it to be a delete request which would delete the alert from the queue(s) it is in.

So in other words, I can figure out how to send the first notification to the first user listed on the policy but I can not figure out how to send the notification a second or third time if they have not resolved the alert.

A bit about my stack, I am using Node.js/Express.js (for HTTP routes) as well as a few AWS services, SQS/SNS/SES. For my database it is MongoDB in a replica set.

Message gotten by alert service from alerts queue

{ "name":"Example website", "policy":"5c8c39de9a77b60a09cadd5c", "url":"example.com", "code":500, "errorMessage":{ "code":"ETIMEDOUT","connect":true } }

​

Example policy

{ id: objectID companyID: '1', name: 'Polciy name', number_people: 3, person_one: 'person_oneID', person_two: 'person_twoID', person_three: 'person_threeID', escalate_time: 5, // Can be up to 30 createdAt: Date.now }

​

Notification part of user model

Note: The values for method can be either email/sms/call and doesn't matter their order or how many are included past 1

Note 2: The contact times can be any value of 10 or under

notification: { first_contact_method: email, second_contact_method: sms, third_contact_method: call, first_contact_time: 1, second_contact_time: 3, third_contact_time: 10 }

​

Improve this question
4
  • 1
    Not a full answer, but looping over IDs is going to be costly unless you are in an environment where alerts will only go to a handful of people/services at most. It also invites concurrency bugs. Commented Mar 21, 2019 at 4:36
  • Basically what you're doing is fanout, and each branch of it may fail. It's best served by a separate queue, with a message per end recipient, and a timing discipline. Commented Mar 21, 2019 at 15:48
  • @Telastyn The idea is that alerts will only be sent to a max of 3 people per alert message I get from my queue. I am not looking to take one message from the alerts queue and send to 100 people. Commented Mar 21, 2019 at 20:21
  • @9000 I thought of that and was going to go with it but the hicup I figured out was what happens when a user resolves an incident? I don't want to still be sending out alerts waking people up at 2am without need be. Also sending more alerts/email/sms has a direct cost to it that scales linearly so if I can only send what is needed that makes sense for my users and for me. Commented Mar 21, 2019 at 20:23

1 Answer 1

Reset to default
3

Sounds to me like an Asynchronous Delayed Task Workflow.

You will need a time delayed queue, and unfortunately some form of shared state.

When the alerting event arrives in your system:

  1. Lookup the information you need to determine who, when, what, etc... to send an alert.
  2. Create a Workflow this acts as a grouping mechanism for the tasks.
  3. Create a task for each "alert" to be sent: address, method, send time, details.
    • If an alert is on a repeating notification, just create the first and set scheduling data in the task.

eg:

workflow: { _id: "...", alarm: true, details: "..." } task: [ { workflow_id: "..." address: "[email protected]", method: "email", time: "2000-01-01T00:00:00", details: "...", schedule: 'once' } , { workflow_id: "..." address: "9876543210", method: "sms", time: "2000-01-01T00:05:00", details: "...", schedule: '5min' } ]
  1. Place the workflow object in a repository somewhere. Ensure that it can be quickly recalled by _id.
  2. Place the tasks on the time delayed queue, time is the time that the queue should reveal the message at.

When a task arrives on the time delayed input queue:

  1. If workflow.alarm == false then task is done, get the next task.
  2. Otherwise send the message by the method.
  3. if task.schedule == 'once' then task is done, get the next task.
  4. Otherwise generate/update a/the task to send at the next time, and add to the queue.

On the messages being sent out have a URL that links back to the alert service. When the person responds, they click the url and get presented with extra details (possibly after signing in). The service can now set worklow.alarm = false, any tasks in the queue will die naturally.

As an aside always limit recurring notifications. If the person hasn't responded on the 255th SMS they probably won't be responding.

I would also point out that sending 255 notifications for 255 simultaneous alerts will be just as irritating. Your responder should be busy dealing with the problems, not clicking links to get the email/sms deluge to stop.

To that end you may want to consider also maintaining a Person: { ..., LastNotified: 'datetime', Responded: true, Workflows: [...] }. When an event comes in it checks the LastNotified and Responded fields to decide what notifications should be sent.

Generally you don't want to notify if they have just responded, they are probably seeing the stream, but maybe in 5-10minutes if they haven't "responded" on the UI to the new events.

When the user respondes, they update thier record, and either update the workflows they are handling, or the task could also check the persons record.

Improve this answer
8
  • Thank you, I am going to see what I can do with this tonight, mind if I ask more questions to you as they come? This is a great reply and is along the lines of what I was thinking just more detailed and better laid out. Out of your reply, I think I understand most of it but not sure what you exactly mean on steps 3 and 4. So say I get a alert/message I would create a workflow object then would figure out what tasks need to be added based off user settings then add that to a queue with say a 60 second delay. Then get from that queue and send an alert. Set a value to true/false Commented Mar 21, 2019 at 20:32
  • When I first get the alert message from my queue I would save a UUID to a DB then with each alert before sending check the DB for a true/false and either send a notification or would delete the message from the queue and DB Commented Mar 21, 2019 at 20:34
  • I have been on google looking for Asynchronous Delayed Task Workflow but not coming across much, happen to have a good resource you can link to? Commented Mar 21, 2019 at 22:33
  • Workflows are an abstraction over a set of tasks. The Tasks are themselves are operated on Asynchronously (one task, several tasks or no tasks of a given workflow might be executing right now), and the tasks are Delayed till a suitable time for their operation. Commented Mar 22, 2019 at 3:20
  • I believe you have got the basic idea. The Workflow is meant to serve as an orchestration point between your need to alert the responders, and the responders need to silence the alert. The tasks which are really the individual notification rules are scheduled to run at the appropriate time. Between when the task was scheduled, and when the task is run, something may have changed - namely the responder has responded. So the first action of the task is to check that it is still needed. Commented Mar 22, 2019 at 3:28

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.