3

How to create alarm when 1) Alert when EC2 instance runs for too long (Say for 1 hour). 2)Alert when number of EC2 instances reaches a threshold (say 5 instances at a time)

One more assumption is, these EC2 instance are specific.Say these alerts applicable to EC2 instances where their instance name start with "test".

When i try to create the alarm , i haven't see this logic in Metrics. Standard Metrics include CPU Utilization, Network In, Network Out etc.

Is there a way to create this alarm either by defining our custom metrics or some other options?

Improve this question
7
  • 1
    Have you explored cloudwatch documentation? Their are apis which allow you to publish metrics Commented Jun 3, 2016 at 14:42
  • @Shibashis had a look into publich metrics.But i am not sure where the logics of the metrics is defined.Eg: docs.aws.amazon.com/cli/latest/reference/cloudwatch/… , i am seing only metric name, statiscal outputs and units are defined. But what i want is, suppose metric name is EC2InstanceHealthDuration (means how long the EC2 instances ran until it started).There should be some unix scripts about the logic that metric name is doing. Please let me know where can i find the same. Commented Jun 6, 2016 at 12:41
  • cloudwatch only collects stats and allows you to create alerts on them, You will need to create scripts for watching how long the instance has been up and then push that metric to cloudwatch. If you dont want to develop such logic you may need to consider other software like datadog,new relic, nagios etc Commented Jun 6, 2016 at 14:03
  • Thanks @Shibashis, so as for the above requirement 1) when EC2 instance runs for too long (Say for 1 hour). 2)Alert when number of EC2 instances reaches a threshold(say 5 instances at a time) doesn't able to use standard metrics. If we write a custom metric, where we have to write the logic of the scripts. I saw in put-metric-data (docs.aws.amazon.com/cli/latest/reference/cloudwatch/…), there is no option to place the scripts. It would be good, if you throw some light on this. Commented Jun 7, 2016 at 10:31
  • For pushing the metric u have many options, i would suggest one of these three options 1> place the scripts on instance themselves 2> create another ec2 where the scripts will run 3> Use AWS lambda scheduled events to push the metrics. Commented Jun 7, 2016 at 13:16

3 Answers 3

Reset to default
2

For automatically deployed instances it’s impossible to setup CloudWatch Alarm as you do not know the instance ID. The only way to setup an alarm was to create an AWS Lambda function that poles all the running instances and compares their launch time to a specified timeout.

The lambda function is periodically triggered by a CloudWatch - Event – Rule.

Use tags to specify different run durations to different machines. For example your launch tool should tag the instance with key value “Test”

Please note this code comes with NO warranties at all! This is more of an example. 

import boto3 import datetime import json from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText ec2_client = boto3.client('ec2') INSTANCE_TIMEOUT = 24 MAX_PERMITTED_INSTANCES = 5 MAILING_LIST = "[email protected], [email protected]" def parse_tag(tags, keyValuePair): for tag in tags: if tag['Key'] == keyValuePair[0] and tag['Value'] == keyValuePair[1]: return True return False def runtimeExceeded(instance, timeOutHours): # Working in to UTC to avoid time-travel during daylight-saving changeover timeNow = datetime.datetime.utcnow() instanceRuntime = timeNow - instance.launch_time.replace(tzinfo=None) print instanceRuntime if instanceRuntime > datetime.timedelta(hours=timeOutHours): return True else: return False def sendAlert(instance, message): msg = MIMEMultipart() msg['From'] = '[email protected]' msg['To'] = MAILING_LIST msg['Subject'] = "AWS Alert: " + message bodyText = '\n\nThis message was sent by the AWS Monitor ' + \ 'Lambda. For details see AwsConsole-Lambdas. \n\nIf you want to ' + \ 'exclude an instance from this monitor, tag it ' + \ 'with Key=RuntimeMonitor Value=False' messageBody = MIMEText( message + '\nInstance ID: ' + str(instance.instance_id) + '\nIn Availability zone: ' + str(instance.placement['AvailabilityZone']) + bodyText) msg.attach(messageBody) ses = boto3.client('ses') ses.send_raw_email(RawMessage={'Data' : msg.as_string()}) def lambda_handler(event, context): aws_regions = ec2_client.describe_regions()['Regions'] for region in aws_regions: runningInstancesCount = 0 try: ec2 = boto3.client('ec2', region_name=region['RegionName']) ec2_resource = boto3.resource('ec2', region_name=region['RegionName']) aws_region = region['RegionName'] instances = ec2_resource.instances.all() for i in instances: if i.state['Name'] == 'running': runningInstancesCount +=1 if i.tags != None: if parse_tag(i.tags, ('RuntimeMonitor', 'False')): # Ignore these instances pass else: if runtimeExceeded(i, INSTANCE_TIMEOUT): sendAlert(i, "An EC2 instance has been running " + \ "for over {0} hours".format(INSTANCE_TIMEOUT)) else: print "Untagged instence" if runtimeExceeded(i, UNKNOWN_INSTANCE_TIMEOUT): sendAlert(i, "An EC2 instance has been running " + \ "for over {0} hours".format(UNKNOWN_INSTANCE_TIMEOUT)) except Exception as e: print e continue if runningInstancesCount > MAX_PERMITTED_INSTANCES: sendAlert(i, "Number of running instances exceeded threshold " + \ "{0} running instances".format(runningInstancesCount)) return True
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

You can use Custom Metric to publish the events in CloudWatch and then you can use that event to set an alarm.

Improve this answer

Comments

-1

I recently implemented a solution (see Github repo) to create alarms for EC2 instances based on runtime, but the approach can also be adapted for instance count. Here's how I approached it:

  1. Create an AWS Lambda function that is triggered periodically by an Amazon EventBridge (CloudWatch Events) rule.
  2. In the Lambda function, use the Boto3 library to describe all running EC2 instances and filter them based on specific tags (e.g., instances with names starting with "test").
  3. For each relevant instance, calculate the runtime by comparing the current time with the instance's launch time. If the runtime exceeds a specified threshold (e.g., 1 hour), send an alert using Amazon SNS or another notification mechanism.
  4. To alert when the number of EC2 instances reaches a threshold (e.g., 5 instances), keep a count of the relevant running instances and send an alert if the count exceeds the threshold.

Here's a simplified version of the Lambda function:

import boto3 from datetime import datetime, timezone def lambda_handler(event, context): ec2 = boto3.client('ec2') # Specify the desired runtime threshold in hours runtime_threshold = 1 # Specify the desired instance count threshold instance_count_threshold = 5 # Get all running EC2 instances instances = ec2.describe_instances(Filters=[ {'Name': 'instance-state-name', 'Values': ['running']}, {'Name': 'tag:Name', 'Values': ['test*']} ]) instance_count = 0 for reservation in instances['Reservations']: for instance in reservation['Instances']: instance_count += 1 # Calculate runtime launch_time = instance['LaunchTime'] current_time = datetime.now(timezone.utc) runtime = current_time - launch_time runtime_hours = runtime.total_seconds() / 3600 if runtime_hours > runtime_threshold: # Send runtime alert send_alert(f"Instance {instance['InstanceId']} has been running for {runtime_hours:.2f} hours.") if instance_count > instance_count_threshold: # Send instance count alert send_alert(f"There are currently {instance_count} running instances.") def send_alert(message): # Implement your alert mechanism here (e.g., SNS, email) print(message)

This Lambda function retrieves all running EC2 instances with names starting with "test", calculates their runtime, and sends alerts if the runtime exceeds the specified threshold. It also sends an alert if the total count of relevant instances exceeds the specified threshold.

Note: Make sure to replace the send_alert function with your desired alert mechanism (e.g., SNS, email).

I hope this helps!

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.