1

While I have worked with AWS for a bit, I'm stuck on how to correctly approach the following use case.

We want to design an uptime monitor for up to 10K websites.

The monitor should run from multiple AWS regions and ping websites if they are available and measure the response time. With a lambda function, I can ping the site, pass the result to a sqs queue and process it. So far, so good.

However, I want to run this function every minute. I also want to have the ability to add and delete monitors. So if I don't want to monitor website "A" from region "us-west-1" I would like to do that. Or the other way round, add a website to a region. Ideally, all this would run serverless and deployable to custom regions with cloud formation.

What services should I go with? I have been thinking about Eventbridge, where I wanted to make custom events for every website in every region and then send the result over SNS to a central processing Lambda. But I'm not sure this is the way to go.

Alternatively, I wanted to build a scheduler lambda that fetches the websites it has to schedule from a DB and then invokes the fetcher lambda. But I was not sure about the delay since I want to have the functions triggered every minute. The architecture should monitor 10K websites and even more if possible.

Feel free to give me any advise you have :)

Kind regards.

Improve this question

1 Answer 1

Reset to default
4

In my opinion Lambda is not the correct solution for this problem. Your costs will be very high and it may not scale to what you want to ultimately do.

A c5.9xlarge EC2 costs about USD $1.53/hour and has a 10gbit network. With 36 CPU's a threaded program could take care of a large percentage - maybe all 10k - of your load. It could still be run in multiple regions on demand and push to an SQS queue. That's around $1100/month/region without pre-purchasing EC2 time.

A Lambda, running 10000 times / minute and running 5 seconds every time and taking only 128MB would be around USD $4600/month/region.

Coupled with the management interface you're alluding to the EC2 could handle pretty much everything you're wanting to do. Of course, you'd want to scale and likely have at least two EC2's for failover but with 2 of them you're still less than half the cost of the Lambda. As you scale now to 100,000 web sites it's a matter of adding machines.

There are a ton of other choices but understand that serverless does not mean cost efficient in all use cases.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

I guess you are right. Seems like I wanted the solution to be serverless but the more I think about it cooking something on ec2 seems like the way to go. Thank you for your input.
I agree with the overall recommendation, but I would like to challenge the Lambda cost analysis. This analysis is only true if a single Lambda instance would only handle a single URL. But if I had to implement this I would do it in Go/Rust and just let the Lambda do multiple calls concurrently. Let's say 10 URLs per Lambda instance. Then you would not have 10.000 Lambda invocations per minute, but 1.000. Runtime would still be more or less the same per invocation (5 seconds), which would slash that cost by a factor of 10. Just something to think about.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.